Progress is lovely

Arts Council England’s (ACE) Creative People and Places (CPP) funding programme aims to “increase attendance and participation in excellent art” in areas of “least engagement”. It recently released two commissioned reports evaluating CPP’s performance. A New Direction (AND) were commissioned to coordinate an evaluation based on meta-analysis produced by Ecorys; The Audience Agency (TAA) were commissioned to produce audience profiling and mapping. Unsurprisingly, self-congratulation ensued, with AND Partnerships Director, Holly Donagh, claiming CPP was producing “powerful results” that could become “a model for the future of arts engagement”.

The trouble is that both reports are fatally flawed. They both rely solely on data from CPP projects and are both produced by organisations with vested interests. Neither report seems to think it might be useful to garner first-hand responses from people who took part in events in these places, nor do they attempt to analyse the perceptions, knowledge and understanding of CPP projects from people in the communities who may not know about or intentionally did not participate in CPP events. This leaves the reports and the data upon which they are based looking inherently biased. Furthermore, AND’s “logic model” and “theory based approach” are both weak and focused solely on outcomes and outputs. This, like TAA’s report, which reduces audiences into flippant categories like ‘Trips and Treats’, ‘Facebook Families’ and ‘Dormitory Dependables’, turns people into “segments” and numbers and pretty little graphs.

How can an evaluation about increasing participation in groups labelled as being low engagers fail to provide sufficient demographic data to enable a detailed analysis of age group, gender, ethnicity and socio-economic background and disabilities/illnesses? The data was often of mixed quality, widely different in terms of proportions of responses, inconsistent and, in some cases missing. And, worryingly, the limited demographic data collected about “previous arts engagement” represented only 19% of total events, and data about “socio-economic background” was collected in only 4% of cases!

The projects did not have a consistent approach to collecting, analysing and disseminating data and, although excellence was deemed particularly important, the 21 projects could not agree on a definition! In almost every case, it was found that the projects did not consider or address ethical issues seriously or at all – they were not following ethical standards; therefore, the meta-analysis cannot adhere to ethical standards. This, when considered alongside the lack of methodological clarity with which the wildly varying data was collected by the various project consortia, makes the data almost impossible to verify and therefore it is deeply unreliable. Indeed, without detailed critical ethnographic research into CPP projects and their wider contexts, the reports reflect a one-sided and probably biased account of CPP which ticks the boxes and leaves everyone singing Hallelujah before quietly sneaking off in the hope the data and methodology is not questioned.

But it is not just the questionable nature of the data that is problematic. The agencies commissioned by ACE to produce the reports means their evaluations are highly likely to be biased. Both TAA and AND are National Portfolio Organisations (NPO). Both are classed as ‘Sector Support Organisations’, with TAA funded from the Grant in Aid budget and AND from Lottery funds. TAA joins the National Portfolio in the 2018-22 round with annual funding of £750k; AND is an existing NPO and will remain so in the next round with funds of £1.5m per annum. And, whilst TAA has a national remit, AND is funded only for work in London.

Furthermore, TAA’s Chair, Sheila Healy, is also Chair of ACE South West Area, a member of Tate St Ives Advisory Council, a member of the HEFCE Catalyst Fund panel, and a director of Situations and Kneehigh Theatre. Another of TAA’s board, Andrea Nixon, is Executive Director of Tate Liverpool, and a board member of the Crafts Council and North-West Museums Group. However, it is even more difficult to accept the decision to commission AND to evaluate CPP because AND is a consortium partner with the Creative Barking and Dagenham CPP project. Interestingly, AND are based in Shoreditch, not Barking and Dagenham.

It is therefore incredibly difficult to accept the neutrality of two NPOs with vested interests in CPP and ACE to produce these two reports about CPP on behalf of ACE – the initiator and funder of CPP. It is therefore of little surprise that the reports celebrate seriously suspect data as “evidence” that CPP is having positive impacts. (The AND report mentions “evidence” 95 times!)

If the arts want to make claims about evidence, then they really should commission some professional, unbiased academic research rather than waste funds on inadequate ‘grey literature’. But these reports function in the same way that hypnopaedia works in Brave New World: if sufficiently repeated, people will accept them as undeniable truths. So, undeniably, progress is lovely. Progress is lovely. Progress is lovely. Progress must be lovely.