Self-study module: Measuring impact
Would you like to focus on impact rather than outputs? Or get to know the main components and principles of impact assessment? This helps to strengthen the sustainability of your organisation and make it future-proof. This module guides you through the how of measuring impact.
When to start measuring impact?
Where possible, evaluation and impact assessment should be planned at the start of your project or activities to help shape them with impact at the centre. Taking an impact-led approach right from the start can help you and your team with how you communicate about your project, attract your audiences, and of course, put in place your data collection plans and workflows. Impact assessment is often confused with evaluation. Yet, they are not the same. Read more about the differences between evaluation and impact.
Preparation
Think about what you want to learn and why. What will you use it for? And whom will you be working with? Will you opt for something light-touch or can you invest a lot of time, energy and money? Aditionally, it's important to formulate what exactly you want to know. Draft research or evaluation questions to help you shape your findings and conclusions, and keep you on track during your data collection
Now, you're ready for the first step. Download the worksheets and write your research questions on the first sheet.
Ensure that you’ll take an ethical approach that respects your audience's privacy and protects their data. Read more about privacy and ethics.
Step 1: Identify your audience
Who will you reach through your activities? And for whom do you want to create and measure change? Your direct audience is that for whom you create (positive) change directly. Your indirect audience might experience positive change because of your activity affecting someone else who they care about, or added value because your activity exists. It’s hard to measure change for everyone, so you’ll need to prioritise. See worksheet 2.
Step 2: Think about the impact you want to create
As a team, think about impact as the big, long-term goal that gives purpose and relevance to your work, the driving force. Like an organisational mission for a more tolerant and inclusive society or a strengthened local economy. It should be something that:
- You can't achieve by yourself
- You are not solely accountable for
- Drives your organisation or your work (like the mission statement or vision)
- Can't be achieved tomorrow, and might take a long time to work towards
- You might not be able to measure your direct contribution to
It’s time to draft an impact statement. Read more in worksheet 3 and add your impact statement to the Change Pathway.
Step 3: Plan outcomes and indicators
Outcomes are changes experienced by your audiences, not actions that you take or results. Think about what your audience might think, feel and do. Get creative through a brainstorm or group workshop and map out the short- and long-term outcomes for your audience. Don’t forget to think about the connections between these.
Some things are always more important than others, and you can’t, and shouldn’t, try to measure everything. Think about what you want to achieve, and prioritise what’s most meaningful about the changes experienced by your audience for you to measure.
Once you have prioritised your outcomes, it’s time to think about how you measure them. Indicators are the measures that help you explore if an outcome has happened.
Indicators can be subjective (a self-reported perception) or objective (an objectively collected measurement). It’s good to get a combination when you are trying to explore if a change has occurred.
The final step with your Change Pathway is to map out your outputs. You use outputs to describe your project in numbers.
Use worksheet 4 to gather information on the topics above.
Step 4: Collect your data
Start with a data-collection plan
Your data is the information that you need to answer your research questions and conduct your analysis. To make a data collection plan, set out:
- For whom will you collect data?
- Where and when?
- How - using what methods?
- How much data should you collect to give you a good and reliable insight into the experiences of your audience?
- How can you build data collection into your existing workflows?
Agree what research methods you can use
Depending on what you want to measure, you can collect data using many different types of methods. Quantitative data are things you can count, qualitative data are stories, opinions etc. Make sure that whatever method of collecting data you choose gets you the type of data you need. Discover different research methods.
Collect baseline and longitudinal measurements where possible
To measure impact, it is useful to try and collect a baseline measurement against which you can compare a later measurement. This shows that change has occurred.
In terms of a longer-term measurement, it’s helpful to assess change not only straight after an activity has occurred but also some time later. This is longitudinal data collection.
Tip! Vergeet niet dat je misschien al bepaalde gegevens hebt. Kijk bijvoorbeeld naar je gebruiksstatistieken op sociale media en betrokkenheid via reacties en het delen van berichten.
Step 5: Analysis and interpretation
Start to describe what you see in your data. What does the data tell you? You can start analysing your data by crunching the numbers using data-analysis tools. For qualitative data, cluster the themes and relationships and important information in your data. You should be asking yourself questions like:
- What does the data tell us in terms of the research questions, indicators and outcomes we defined?
- What does this mean in terms of what we already know or assume?
- What doesn't it tell us and what do we still not know?
- Does this sound plausible and why?
- Is there anything unexpected?
Step 6: Write and visualise your impact story
Make your data make impact for you! It’s now time to write and visualise your impact story.
‘Data storytelling is a structured approach for communicating data insights, and it involves a combination of three key elements: data, visuals, and narrative.'
Your impact story is only just beginning when you present your findings in charts and graphs. An impact story is built on your analysis and interpretation of your data. In it, you will clearly articulate what you learned as well as why it’s important and what you’ll do with what you learned. This is a creative process where you need to think about who you are sharing this information with and what you want to achieve.
This holds true for data visualisation, too. It’s not just about presenting data but explaining what the chart shows and why it’s important. Remember - the less positive results play just as important a part in your impact story, just like the plot twist in a traditional story.
Use worksheet 5 for help on how to:
- Set the scene with your objectives and impact goal
- Introduce your audience as your story's characters
- Set out your plot in terms of your main findings
- Introduce your data in an objective way through plot twists
- Reflect on what it means
Next steps
In the preparation section of this module, we asked you to think about what you wanted to achieve through impact assessment and evaluation. Did you want to make your digital offer better for your audience? Did you want to communicate your impact more effectively to funders to support more sustainable funding? Did you want to increase the amount of budget allocated to digital culture in the school you’re working with? Put your intentions into action and create more impact!
More impact?
Use the insights and experience from this module and impact assessment in your next activities. Embed thinking about impact in everything you do. Curious to learn more about different impact measuring methods? Learn from DutchCulture and Festival Cement.
