Storytelling – a different way of evaluating

Let me start by telling you a story about us.

Arts at the Old Fire Station (AOFS) is an arts centre in Oxford, where we programme and create art that takes risks, asks questions and entertains, as well as supporting early and mid-career artists. What makes us different from other art centres is that we also share our building with the homeless charity Crisis. We work closely with Crisis members – people with experience of homelessness – in every aspect of our work – in creative projects, programming, ushering shows, and wider decision making.

Our unique relationship with Crisis means we offer a public space that is shared by very different people and helps to break down barriers. It’s a place where people are able to define their own labels – you might walk through our doors as someone experiencing homelessness, and leave as a volunteer, trainee or trustee at AOFS.

Ever since we opened in 2011, we’ve been interested in understanding what impact our work has on the people involved – whether artists, Crisis members, staff, volunteers or audience members. We’ve also had to be able to demonstrate that impact to the funders that make our work possible.

Drawbacks of conventional evaluation

Over the years we’ve experimented with different ways of monitoring and evaluating, but we’ve found it a struggle. More conventional approaches to evaluation were problematic for a number of reasons:

• They risked undermining the relationships we had built with people

We work hard to build relationships based on what people can bring – what are their skills and interests? Are they creative? What role could they play in helping us run our art centre? We don’t ask people about their pasts, or about their housing situation. Trying to evaluate impact by asking people to fill out forms and take part in interviews risked undermining this relationship we had built – it felt reductive and inappropriate.

• They were a distraction

We’re an art centre and our focus should be creating and supporting great art and involving people in this process. Our preoccupation with monitoring, capturing and analysing data interrupted the creative process, and took energy and focus away from the things that mattered.

• They were incomplete

The number of people who attend a photography workshop is just a number. It doesn’t tell us about the quality of the session or what it meant to the people who attended. Counting numbers alone risked overstating the significance of the data we captured. Similarly, measuring impact against pre-determined outcomes meant that change, and what it looked like, was decided by us. Impact that didn’t align with these outcomes was in danger of going unrecognised.

This is not to say that these evaluation methodologies do not have their place. Nor is it to say that we don’t still count numbers – we do. But they were incomplete, an ‘add-on’ and risked being disruptive to our work, so we decided to try something new.

The Storytelling Methodology

In 2017, with the support of external evaluators Anne Pirie and Liz Firth, we piloted the storytelling methodology – based on the Most Significant Change (MSC) technique. It involves collecting stories of significant change from participants, and the participatory interpretation of these stories.

Unlike conventional approaches to monitoring, MSC doesn’t use quantitative indicators developed in advance. Rather, the storytellers decide on what the most significant impact is for them. MSC is good for measuring change that is intangible or fuzzy – unexpected, emergent, personalised or diverse – and understanding how change happens.

Building on MSC, we began using storytelling. The methodology involves the following key stages:

1. We recruit a team of story collectors, including staff, volunteers, Crisis clients and artists.
2. The story collectors meet individually with the storytellers – the artists, staff, Crisis clients and volunteers who had participated in the activity being evaluated. They have a conversation about what changed for them personally, why this change was important, and the main things that made it happen. Crucially, this isn’t an interview, it’s a relaxed, informal conversation on equal terms.
3. This conversation is recorded, transcribed and then edited down into 1-2 page stories which aim to retain the teller’s ‘voice’, reflect their insights, and capture the reader’s attention.
4. We hold a discussion day, which is attended by staff, Crisis clients, volunteers, artists, trustees and partners. The group read and listen to the stories and then come together to discuss them in a facilitated meeting, which aims to help understand the themes and learning emerging from the stories and how they can help AOFS develop its work.
5. We share the stories publicly and write a report about the learning which emerged from this process.

Storytelling in practice

We recently used storytelling to evaluate ICON, a collaborative photography project between AOFS and Crisis which was led by internationally renowned photographer Rory Carnegie. The project brought together Crisis clients to work with Rory to recreate iconic British photographs. There was buzz around ICON, and everyone involved knew it was a special project. But what did it mean for people on a personal level? What was its impact? What was it that made the project resonate and work for the people involved? To help us explore this we collected stories from 11 people, including staff, artists, and Crisis clients. Here are some of the things they said:

In the photo I was Emmeline Pankhurst. And yes, it gave me another life. Yes. I feel that I have another life. Pankhurst, Pankhurst opened for me, really opened my eyes. Pankhurst made my world get wider.

What does this mean I can do, if I’ve done this? I’m sure there’s more sides to me that I didn’t even know about. I need to explore that because I haven’t been happy before. I need to see what else is out there.

I genuinely felt that this could be a beginning of a great situation in my life, around other people’s great situations in their lives. Through such a simple thing as photography.

The stories were rich and insightful. People spoke about what the project had meant for them personally: they had developed technical skills, confidence, a sense of belonging and discovered new things about themselves. People also spoke about what led these changes to happen – high expectations, collaboration, a break-down of hierarchies, the degree of both flexibility and structure in the way the project was run.

We learnt a great deal from the ICON stories about what change looks like for the people with whom we work, and what enabled that change to happen. The process of storytelling was fun and creative, and the stories themselves had a lot of creative potential. They were the basis for a short piece of theatre, and we intend to use them as part of the ICON touring exhibition in 2021.

Why Storytelling has worked for us

This is the story of our relationship with storytelling so far. It’s offered a huge step up from the evaluation approaches we’ve used in the past, giving us a better understanding of the very personalised, diverse outcomes that participants experience. Rather than us determining outcomes, it enables participants to tell us what change looks and feels like for them. It’s holistic and human, and it allows us to understand change in relation to the wider context of a person’s life. The process itself is also enjoyable, meaningful, and collaborative.

Storytelling has shifted evaluation from being an ‘add-on’ at the periphery of our work, to something fundamental to who we are and what we do. It has enriched many other aspects of our work, enabling us to bring to the fore the voices of the people we work with in a way that feels real, engaging and human.

We are still experimenting with storytelling, and each time we use it we learn more about what works and what doesn’t. We are currently in the midst of a storytelling project in partnership with local organisation Oxford Hub, collecting the stories of people involved in Oxford’s community response to Covid-19. We’re learning a lot about how to collect stories over Zoom, and how to use the methodology in collaboration with an external partner.

While we are still working out how the methodology can be developed further, for now we feel we have happened upon a way of learning which is exciting, meaningful, and creative.  

Sarah Cassidy is Inclusion Manager at Arts at the Old Fire Station
https://oldfirestation.org.uk/about/our-stories/