ACE denies statistics are misleading as regulator demands more clarity

The UK Statistics Authority (UKSA) has backed concerns raised by ArtsProfessional about whether a huge drop in visits to exhibitions at National Portfolio Organisations (NPOs) reported by Arts Council England (ACE) is in fact largely down to weak methodology.

A public letter from Ed Humpherson, head of UKSA’s regulatory arm, castigates ACE for an array of statistical misdemeanours that fail to meet the official Code of Practice, and which led to a “lack of clarity” in the presentation of visitor numbers in the funder’s latest annual report.

But ACE still denies that its presentation of the figures was misleading, and says it has no plans to update the report.

Public art

In its annual report for 2017/18, ACE reported a drop in exhibition visitor numbers of 16.6m between 2015/16 and 2016/17 at NPOs and Major Partner Museums across England. This figure, equivalent to almost three times the British Museum’s annual visits, included a drop of 17m at London organisations, which was marginally offset by increases in other parts of the country.

The funder said the reasons for the change included “the temporary closure of a number of significant cultural buildings for capital work” and “a number of biennial events that took place in 2015/16 and not 2016/17”.

However, AP suggested that the apparent trend may have arisen largely because UP Projects, an NPO with exceptionally high visitor numbers due to the public locations of its exhibitions, appears to have been included in ACE’s calculation of audience numbers for 2015/16, but not 2016/17.

This would provide a plausible explanation for the majority of the reported fall. But ACE refused to confirm which organisations had been included and excluded in its samples, citing commercial confidentiality.

Serial misdemeanours

The statistics watchdog has now confirmed that the concerns raised by AP are valid, and listed a series of important aspects in which the funder’s statistical practice was unsatisfactory.

Humpherson told Andrew Mowlah, ACE’s Director of Research, that: “we judge that there is a lack of clarity around some aspects, including which organisations are excluded from the official statistics, what constitutes an ‘extreme response’ and cut-off points for defining outliers.”

“It seems unclear where it is appropriate to make year on year comparisons and where you advise that this should not be done. Additionally, it is not easy to reproduce the official statistics from the raw data spreadsheets that are also published.”

Humpherson concluded his letter by underlining that the regulator “share[s] the concerns raised by the correspondent around the lack of clarity provided in the methodology and about the uses of these statistics”.

ACE welcomes statistical training offer

In his response, Mowlah said “audience and visitor numbers are difficult to collate accurately, as you acknowledge in your letter.”

He added: “We strive to be as clear as we can when we spot anomalies, or where we’ve made decisions about outliers, but it is not possible for us to list those outliers in more detail without naming specific organisations.”

ACE, which named specific organisations in the original explanations for the trend it offered to AP, still refuses to confirm that UP Projects was included in the calculations for 2015/16 and not for 2016/17. Nor has the funder explained why there might be any legitimate reason for doing this, when UP Projects operated normally and submitted its visitor figures across both years.

An ACE spokesperson told AP: “We don’t comment on individual organisations and as you can see from our letter, there might be a variety of reasons an organisation might be in the analysis one year and not the next.”

They added: “We strive to present our data in a clear, concise and transparent way. We do not believe our NPO data for 2017/18 was misleading, and we are very pleased to hear that in future we are going to be able to put more explanatory notes on future data releases. We do not have any plans to update our annual report”.

However, in his letter, Mowlah says ACE would be happy to meet with the Government Statistical Service’s Good Practice Team to receive guidance on how to better present statistics in future – as recommended by the regulator.

He told Humpherson: “We recognise that we are operating in an increasingly data driven and data aware society, and as such the need to present and explain data clearly is of paramount importance.”

Performance indicator

Sustained attendance and visitor numbers at NPOs and Major Partner Museums are critical to ACE, as they have for many years been among the Key Performance Indicators (KPIs) set out in its management agreement with the DCMS.

They are included among seven KPIs in ACE’s latest management agreement with DCMS, which runs from 2016-2020. These were approved in May last year. A spokesperson previously told AP that “there are no numerical targets – the idea being that reporting against them will help to establish a baseline for future Management Agreements.”