Green light for artistic quality measurement scheme

Arts Council England (ACE) is finally moving forward with the quality measurement system it will be forcing its National Portfolio Organisations (NPOs) to use to evaluate and benchmark the impact of their artistic work.

Mandatory for ACE’s larger NPOs (band 2 and 3) – including those that already implement sophisticated audience evaluation programmes of their own – the new Impact and Insight Toolkit will be used to capture audience responses to a standardised set of pre-determined statements known as ‘quality metrics’.

A supplier has been chosen to deliver a digital platform, which will be rolled out in October and which NPOs will be required to use to report on their data outcomes from April 2019.

The controversial scheme was first proposed by a group of arts organisations in Manchester in 2012, and further developed on the “working assumption” that it “must aim to lessen the reporting burden” on funded arts organisations.

ACE is pushing ahead with it despite sector concerns about the approach it takes to assessing quality and the administrative burden it will place on arts organisations.

Universal adoption

256 NPOs will be required to use the toolkit to complete four evaluations each year, which will involve collecting self, peer and public feedback. The aim is to find out how people rate the quality of the artistic work that they have experienced.

Whist it is not yet compulsory for band 1 NPOs or Sector Support Organisations, ACE is expecting most of them to opt into the scheme. ACE Chair Nick Serota sees the system as being especially valuable for NPOs “in a small town or rural area [where] you might be the sole cultural focal point of the community”.

Two-thirds of band 1 NPOs and SSOs are expected to register voluntarily to use the platform by September 2020 and over half to be completing at least one evaluation per year by September 2022.

Quality measurement

The £2.3m four-year contract to deliver the system has been won by Counting What Counts Ltd, the company that ran ACE’s £300k pilot of the scheme in 2015.  

Originally known as ‘Quality Metrics’ and subsequently ‘Consumer Insight Toolkit’, the system has now been launched under the name ‘Impact and Insight Toolkit’. An ACE spokesperson told AP: “The name was changed in consultation with Counting What Counts as we wanted one that explained more clearly the aims of the platform.”

But despite its new name, ACE Director of Research Andrew Mowlah has confirmed that the aim of the toolkit is still to measure perceptions of artistic quality. He said it differed to Audience Finder, the ACE-supported insight tool provided by The Audience Agency, which analyses audience demographics, behaviours, motivations for attending and quality of experience.

“This project focuses on how people rate the quality of the work that they have actually seen,” he said.

Procurement turmoil

Counting What Counts Ltd was first identified as the preferred bidder to deliver the programme over a year ago, but the original tender was withdrawn shortly afterwards, as ACE was forced to re-run its procurement process when it was revealed that there had been irregularities in the initial process.

94 requests were made for the new tender documents, but only three companies submitted bids for the contract. Counting What Counts Ltd was widely tipped to win, given that the earlier stages of the quality metrics pilot project were led by its Director John Knell, and ACE had already ploughed £700k into developing and piloting the system, including the company’s proprietary digital platform, Culture Counts.

ACE will not, at the end of the four-year contract, hold any ongoing rights to use the Culture Counts platform or database. If its contract is not renewed, Counting What Counts Ltd will have to “enable the smooth transfer of data and other intellectual property developed” to ACE and/or another service provider.

The tender document stipulated that the supplier “shall not use the data collected for commercial benefit”, but Counting What Counts Ltd will be permitted to charge NPOs for additional bespoke services.

Owen Hopkin, the new Director of Audience Insight and Innovation at Arts Council England, defended the selection of Counting What Counts Ltd, saying they “demonstrated a high level of innovation in their bid and extensive background knowledge and experience of working with the arts and cultural sector”. He added they “gave confidence in their ability to build understanding and support for the toolkit amongst the organisations we work with and to develop the culture of self, peer and public review we want to achieve through this work”.

Unresolved issues

Work is starting to finalise the details of the system and efforts will begin next month to “re-engage the sector,” so that the larger NPOs are able to start using the system from October 2018.

Counting What Counts Ltd will be expected to “work with the sector to deliver ongoing development and improvement of the toolkit to ensure it remains relevant and meets the wide-ranging demands of NPOs and ACE”.

Work is still progressing to make the quality metrics questions fit for purpose and accessible for use among children and young people, people with profound and multiple learning disabilities, and people with English as their second language. Mowlah said: “I think we are making some really good progress in trying to innovate the way the Metrics are discussed and talked about actually so that it is truly reflective and accurate about the way people would describe the work that they are seeing.”

It is yet unclear how the necessary “ongoing refinement of the existing quality metrics” will be compatible with the aim of benchmarking across all NPOs.

PR exercise

Given the widespread scepticism – and some outright hostility – to the concept of quality measurement for artistic work and the potential workload involved in implementing the new system, the tender also required the supplier to deliver “a programme designed to build renewed support, confidence and understanding for the toolkit”.

This will include recruiting an initial group of “sector champions”, building interest in the toolkit from band 1 NPOs and SSOs to prompt take-up, as well as producing and disseminating “useful thought pieces and engagement material for the NPO cohort and the wider sector”.

ACE has begun its own charm offensive by publishing a recorded podcast in which Andrew Mowlah and its Chief Executive Darren Henley respond to questions and discuss the pilot initiative with three senior arts professionals.

Policing

Counting What Counts Ltd will be required to report back to ACE on the extent to which NPOs are complying with their contractual requirement to participate in the scheme. It will provide ACE with ongoing monitoring information, telling them which NPOs are using the system as intended and reporting on their survey response rates.

ACE has also specified that the toolkit must be configured to provide it with access to “aggregate and individual NPO data in real time throughout the duration of the contract”. A clause in band 2 and 3 funding agreements requires NPOs to comply with this. This will enable ACE to bypass NPOs and take the data it wants directly from Counting What Counts Ltd.

To comply with GDPR data legislation, Counting What Counts Ltd will have to enter into separate data arrangements with each NPO, making it clear that it is NPOs that control the data they collect, including the right for it to be deleted.

Were they to request any such deletions however, it is yet unclear how this would be compatible with their contractual obligations to ACE. An ACE spokesperson told AP: “The provider and NPOs will need to ensure compliance with GDPR, their contract and funding agreements.”

9/7/18 A previous version of this article incorrectly stated that there were two bidders for the contract, not three. Also, Arts Council England has asked us to point out that not all of the £700k it spent on the quality metrics pilot was invested in the development of the Culture Counts platform.