“Only a fool or a knave” trusts quality metrics, say academics

Metrics-based approaches to assessing cultural value “invite political manipulation and demand time, money and attention from cultural organisations without proven benefit,” according to a group of academics involved in a research project examining the value of culture.

In a new paper, published in the academic journal Cultural Trends, researchers deliver a damning assessment of the type of standardised system for measuring artistic quality that Arts Council England (ACE) is currently preparing to make compulsory for many of its National Portfolio Organisations.

Their paper, ‘Counting culture to death’, refutes the “widely held belief” that “a set of numbers can provide vindication, or at least insurance, in the constant struggle to justify public funding”. They conclude that attempts to quantify cultural value are not delivering on their promises, and bring “destructive” unintended consequences.

The paper states that using indicators and benchmarks to assess cultural activities, “which exhibit no obvious capacity for scalar measurement”, is a “political act”. The “ostensible neutrality” of this approach is, they say, “a trick of the light trying to launder responsibility for judgment in the competition for scarce resources”.

The research, conducted by Robert Phiddian, Julian Meyrick, Tully Barnett and Richard Maltby from Flinders University in Adelaide, was funded by the Australia Research Council as part of the ‘Laboratory Adelaide: the Value of Culture’ project, which is examining issues around the assessment of culture.

Culture Counts

The researchers focused on the use of Culture Counts, the metrics-based assessment system that originated in Western Australia and was adopted by the state government as part of its Public Value Measurement Framework. It is the same system that ACE used in its pilot Quality Metrics scheme in 2015/16, which involved 150 arts organisations across England.

The researchers warn that the effects of using Culture Counts in Australia “are worth considering in places pointed in the same direction, such as the UK”. Following trials in Australia, the state of Victoria has decided not to continue with the system, and the Australia Council, the national cultural agency, does not intend to implement it either.

The research findings add further fuel to controversy over the forthcoming roll-out of this scheme across England, participation in which will be a condition of funding for larger National Portfolio Organisations from 2018.

The scheme will see arts organisations ask audiences to rate their artistic work on a numeric scale in relation to a set of standardised metrics of quality, and triangulate this data with self and peer assessment.

Despite concerns expressed by many who took part in the pilot in England, a £2.7m contract to deliver the Quality Metrics scheme has been put out to tender. Although ACE has not yet committed to using the Culture Counts system, it is unclear how any other system would be able to meet the criteria for delivering it.

Misguided beliefs

The researchers describe Culture Counts as: “A sophisticated and entrepreneurialised version of the misguided belief that everything that matters in culture can be counted, and that doing so will secure the sector’s social and political base.” The metrics it uses are, they say, “essentially marketing analytics rather than a window on artistic value” and “there is simply no numerically valid way” of valuing one artform over another.

They continue: “…only a fool or a knave will claim [two different artforms] can be assessed comparatively without making use of a critically informed judgment.”

Attention is also drawn to the potential misuse of quantitative data generated in this way. They point out: “It is politically naïve to think that metrics such as Culture Counts can be quarantined for practitioners and policy officers who will use the numbers with care, as indicators of well-understood value-generating creative processes.” Rather, they will be “used as weapons, both of attack and self-defence. Artists competing for funds to realise a vision will not resist the temptation to trumpet ‘a 92 in the ACE metric’ any more than vice chancellors have refrained from boasting of their rankings in university league tables.”

ACE was due to announce the winning bidder for its Quality Metrics contract in April, although this has been delayed. A spokesperson told AP: “Due to pre-election sensitivities we have been unable to issue the decision letters.” These are now expected after 8 June.