Economic Impact Assessments too varied for useful comparisons, says report

The lack of consistent methodology across economic impact assessments “reduces the potential to make meaningful comparisons” between studies in the cultural sector, a major review of UK and overseas assessments has concluded.

Although it finds that economic impact assessments are overall “fit for purpose”, it criticises the limited exploration of key economic concepts and calls for future economic impact assessments to seek a more “systematic consideration” of the consequences of investment.

The report – ‘Local economic impacts from cultural sector investments’ – is based on a literature review of economic impact assessments of arts, heritage, museums and galleries, cultural events. It was commissioned by the DCMS and completed in May 2014 by consultants ECORYS, but only released to AP under Freedom of Information last month following requests dating back to 2015.

It was written two years before a report by the Arts and Humanities Research Council (AHRC) into cultural value, which concluded that economic impact studies suffered from poor research practices and “rarely rested on rigorous analysis”. 

Weaknesses

The report highlights some specific weaknesses that undermine the usefulness of economic impact studies. For instance, it finds that ‘leakage’ – spending outside of the area of focus for the economic assessment – is usually calculated in such a way as to “exclude the expenditures which leak outside the area of analysis”, such as by staff paid by an investment who live (and spend) elsewhere.

Equally, it finds it is “not always clear” if and how adjustments have been made for ‘deadweight’ – the extent to which the measured impacts would have been attained even without investment. The studies reviewed have also, it says, “implicitly assumed displacement to be zero”, but notes that there are circumstances in which increased spending in one location could be funded by reduced spending somewhere nearby.

Missing elements

While the report accepts that economic impact assessments take place in different contexts, with varying access to resources and pre-existing data, it finds there is a “limited exploration” of wider economic effects, such as the role of culture in improving the attractiveness of a location to live and work.

It also states that none of the studies under review had made a comprehensive assessment of wider impacts, such as environmental impacts.

“Including at least some recognition of wider social, cultural and economic outcomes is desirable and, in some instances, may help to strengthen the economic case,” the report concludes.

Recommendations

The report recommends that steps to assess and evaluate economic impact are built in from the outset of any significant investment in cultural assets or infrastructure. It says adjustment for additionality – the extent to which something happens as a result of an intervention – “should be considered essential” for each area of expenditure and impact assessed.

To support this, the report provides a checklist to inform planning and guide the ‘systematic consideration’ of different aspects of additionality. These involve:

• Temporary effects – related to specific investment, which will result in a temporary effect on employment and a one-off impact on Gross Value Added
• Direct effects – impacts arising from the employment of staff and purchase of goods and services from suppliers
• Indirect effects – arising from visitor activity drawn to the site, bearing in mind a proportion would be expected to spend their money in the local area anyway
• Wider economic effects – the impact of cultural facilities on the image and perceptions of an area, such as attractiveness as a place to live and work

Furthermore, while it accepts that estimating additionality adjustments is not always possible, it encourages researchers to source appropriate, evidence-based adjustments from existing literature.

The DCMS has said it will publish the report in due course. In the meanwhile, the full report can be read on the link below.