Evaluation Principles: Robust
Find out more about adopting rigourous and appropriate evaluation approaches and methods.
Robust
Rigorous
Open-minded
Proportionate
Introduction
In order to inform genuine change within the cultural sector, we need to be confident enough in the findings of our evaluation activity and approach.
Part of the skill of conducting robust evaluations involves balancing expertise and skill with openness to unexpected outcomes or viewpoints.
Taking a robust approach to evaluation involves the appropriate and rigorous application of different methods, to not just find out ‘what’ happened, but understand in more depth ‘why’ and ‘how’. This approach can help us learn from our failures along the way.
Principles into practice: questions to consider
We have designed the following questions to help you think through how the Robust Principle can apply to your specific context of practice.
Some questions may be more relevant to you than others. Consider which might help you spark productive discussions with colleagues, partners or stakeholders.
Is your evaluation activity … rigorous?
- Are you confident about what the available evidence does – and does not – allow you to say?
- How is your evaluation activity grounded in empirical research (i.e. not all theoretical but grounded in experience, observation and practice)?
- How do you decide which methods are most appropriate for evaluating different projects or programmes?
- Do you clearly describe your methods so that they might be replicated? (This might include how focus group participants were selected, how many survey responses you collected or how you coded or filtered your data.)
- Could you explore mixing methods in your evaluation activity to enrich your understanding (e.g. combining qualitative and quantitative research approaches)?
- Does your evaluation activity consider different types of data (i.e. not all numbers or opinions/anecdotes)?
Is your evaluation activity … open-minded?
- What changes can you make to your evaluation activity to incorporate unintended outcomes and new viewpoints? Where are the underrepresented narratives of value in your work?
- How might you make room in your evaluation activity to learn from your failures as much as communicating and celebrating your successes?
- How does your evaluation activity allow you to learn from others and critically engage with your own assumptions?
- Do you make an effort to listen to evidence that might contradict your beliefs and/or lived experience?
- Are there ways you can make it easier to hear and act on critical feedback, without being defensive or dismissive?
- How could your evaluation be carried out with more independence, regardless of outcome?
- Is your evaluation activity too focused on prescriptive, tried-and-tested methods? What is stopping you from trying something innovative, which might open up a new way of learning or working?
- Do you need additional training to help you refresh your knowledge of evaluation methods and techniques and best practice?
Is your evaluation activity… proportionate?
- Does your evaluation activity provide you with a good return on your investment (e.g. of time and money), achieving its original aims? If not, why not?
- Do you need to evaluate all of your work? What evaluation activity provides you with the greatest potential learning?
- Have you done this evaluation before? Are there ways in which you could take a more pragmatic approach, such as reducing the scale or scope?
- Are there others who are interested in the same questions with whom you could collaborate and share resource?
- Can you access additional funding for external evaluation support?
- If your evaluation activity is designed to meet certain funding criteria, can you discuss with your funders what might be most useful, in terms of types of information and format of reporting? Can you ask them to provide examples of the type or level of evaluation for different funding amounts?
Related resources
Hosted by Chuck Blue Lowry, this episode of Reflecting Value: Evaluation Principles in Practice explores the proportionality principle, and how we can go slower in our evaluation work.
Rachael Disbury at Alchemy Film & Arts shares how the organisation changed their approach to audience engagement, paying close attention to what wasn’t working and creating more tailored approaches for engagement as a result.
Stephanie Pitts shares four recommended reads to help deepen your understanding of audiences, audience engagement and audience research.
My essential reads: cultural value and evaluation
Beatriz Garcia shares five recommended reads to help deepen your understanding of approaches to evaluation.
Since the Covid-19 pandemic there has been much attention on children and young people’s mental health and wellbeing. But where do arts and cultural programmes fit into this picture? This research digest explores the evidence to provide a snapshot of the current thinking.
