Measuring the difference you make

There are lots of reasons why arts organisations might want to measure the difference they make – their social impact. It might be because a funder or commissioner requested it, or because they’re trying to make the case for funding in the first place. Or they’re designing a new programme and they need to know what works, or they’re trying to get politicians to back a cause.

A more developed theory of change can help to identify where evidence backs up your assumptions

These are just some of the reasons given by the organisations we at Alliance for Useful Evidence spoke to in our recent Evidence for Good report that explores how charities of all shapes, purposes and sizes use evidence in different ways.

Centres for evidence

Whatever the motive, at the Alliance for Useful Evidence and Wales Centre for Public Policy, what matters is that the evidence used is high quality, independent and robust. We are committed to improving the use of evidence in decision-making, whether that’s by policy-makers or practitioners.

We adopt a fairly broad definition of evidence, ranging from academic research and evaluations to expert knowledge. At the Alliance – an open access network of over 3,800 individuals from across government, universities, charities, businesses, local authorities and public services – we run events, share ideas and deliver training and support on using evidence effectively.

At Wales Centre for Public Policy, an independent centre based at Cardiff University, we support ministers and public services to access rigorous independent evidence about what works, and contribute to the evidence base on what works in evidence-informed decision-making.

Payment by results

We know there’s growing pressure on organisations to demonstrate social impact. It’s an essential part of the increasingly common outcomes-based approach to commissioning, such as the Payment by Results (PbR) system, where organisations are only paid if they deliver an agreed set of outcomes.

While there’s a real opportunity here for innovation and creativity, PbR can be a challenge for small organisations that simply can’t afford to take big risks. Social Impact Bonds (SIBs), where investors take on that financial risk, go some way to addressing this challenge. So do creative approaches like the Welsh Government’s Innovate to save programme, run in partnership with Nesta’s Y Lab and Cardiff University, which supports organisations to try out new approaches and measure their success.

Using theory of change

So how do you measure impact? Whatever kind of organisation you are, the first place to start is with a theory of change – at its most basic level, a description of how you think your programme achieves the outcomes you’re aiming for. There are lots of resources freely available online, produced by the likes of Nesta and NPC, to help organisations develop their theories of change.

A more developed theory of change can help to identify where evidence backs up your assumptions, and where there are evidence gaps – gaps that you might want to fill with your own research. The next steps are working out how you can go about filling those gaps, and this is where a horses-for-courses approach makes most sense.

Do you want to prove whether your programme works for a certain group of people? You’ll need an experimental design, like a randomised controlled trial. That’s what Remember a Charity, a legacy-giving network and campaigning organisation, did to see what kind of language worked best in asking people to leave money to charity in their will.

Or are you interested in why a programme works, and how? Qualitative research methods like interviews and focus groups can provide that kind of insight.

Of course, not every organisation is in a position to conduct the right kind of research themselves. The charities we spoke to for our Evidence for Good report had some helpful advice for organisations looking to be smarter about the way they use evidence – particularly small charities with limited resources. Most of the advice revolved around the importance of relationship-building, such as making connections with academics at local universities who are keen to deliver real-world impact with their research. This resulted in Masters, PhDs and shorter pieces of robust academic research on the charities’ work.

Emma Taylor-Collins is Senior Research Officer at the Alliance for Useful Evidence and the Wales Centre for Public Policy. www.alliance4usefulevidence.org
www.wcpp.org.uk
Tw @ETaylorCollins
@A4UEvidence
@WCfPP

The Alliance for Useful Evidence is currently working with the National Council for Voluntary Organisations on the use of evidence in campaigning and would like to hear from arts charities that have examples of good practice or challenges they've come across: email [email protected]