When do you measure impact?
Plan an evaluation and impact analysis at the start of your project or activity whenever possible. This allows you to place impact at the core of what you do. But evaluation and impact are not the same. An impact-driven approach from the beginning of your project can help you and your team with project communication, audience engagement, and, of course, setting up your data collection plans and workflows. The earlier you start, the more you gain.
Preparation for this module
Think about what you want to learn and why. What do you want to use it for? And with whom will you work on this? Will you take a light approach, or can you invest a lot of time, energy, and money? Additionally, it is important to note what exactly you want to know. Formulate research or evaluation questions to better articulate your findings and conclusions and to stay on track while collecting data.
You are now ready for the first step. Download the worksheets; on the first worksheet, you can immediately write down your research questions.
Are you conducting impact research? Ensure an ethical approach that guarantees the privacy of your audience and protects their data. Read more about data privacy and data ethics.
Step 1: Determine who your audience is
Who do you want to reach with your activities? And for whom do you want to create and measure change? Your direct audience is the audience for whom you bring about (positive) change directly. Your indirect audience may experience a positive change because your activity influenced someone they care about or added value because your activity exists. It is challenging to measure change for everyone, so you will need to set priorities. Use the diagram on worksheet 2.
Step 2: Determine what impact you want to make
As a team, consider impact as the big long-term goal that gives meaning and purpose to your work, as the driving force. For example, the mission of an organization for a more tolerant and inclusive society or a more robust local economy. Impact is something:
- That you cannot achieve alone
- For which you are not solely responsible
- That drives your organization or work (like a mission statement or vision)
- That cannot be achieved tomorrow and may take a long time to work towards
- To which your direct contribution may not be measurable
It is now time to draft an impact statement. Continue with worksheet 3 and add your impact statement to the Change Pathway.
Step 3: Create a plan
Outcomes are changes experienced by your audience, not activities you perform or results. So, create a plan for the outcomes you want to achieve and the indicators you can measure. Think about what your audience might think, feel, or do. Organize a brainstorming session or group workshop and map out the short- and long-term outcomes for your audience. Don’t forget the connections between these two types of outcomes.
Some things are always more important than others, and you cannot measure everything, nor should you try. Determine what you want to achieve and then identify which changes experienced by your audience are the most important to measure.
Once you have determined which outcomes are the most important, it’s time to think about how you can measure them. Indicators are the benchmarks that allow you to determine whether a specific outcome has occurred.
Indicators can be subjective (a self-reported experience) or objective (an objectively obtained measurement). A combination of these two is a good idea if you want to determine whether change has occurred.
The final step in your Change Pathway is to map your outputs. Outputs allow you to express your project in numbers.
Use worksheet 4 to map out the components above.
Step 4: Collect your data
Create a data collection plan
Your data forms the information you need to answer your research questions and conduct your analysis. To create a data collection plan, you need to determine:
- From whom you want to collect data?
- Where and when?
- How, using which methods?
- How much data you need to collect to get a good and reliable picture of your audience’s experiences?
- How can you integrate data collection into your existing workflows?
Decide which research methods you can use
Depending on what you want to measure, you can collect data using many different methods. Quantitative data are data you can count, while qualitative data are stories, opinions, etc. Ensure that whatever data collection method you choose, you obtain the type of data you need. Read more about research methods.
Collect baseline and longitudinal measurements where possible
To measure the impact of something, it is useful to try to conduct a baseline measurement that you can compare with a later measurement. This indicates that a change has occurred. For the long term, it’s a good idea to measure change not only immediately after an activity but also some time later. This is longitudinal data collection.
Tip! Don’t forget that you may already have certain data. For example, look at your social media usage statistics and engagement through comments and shares.
Step 5: Data analysis and interpretation
Start by describing what you see in your data. What can you infer from the data? You can analyze your data using data analysis tools. For qualitative data, it is important to bundle the themes, connections, and key information in your data. You should ask yourself questions like:
• What do the data tell us about the research questions, indicators, and outcomes we have identified?
• What does this mean for what we already knew or assumed?
• What information do the data not provide, and what do we still not know?
• Does this sound plausible, and why?
• Is there anything unexpected?
Step 6: Write your impact story
Ensure that your data leads to impact for you! It is now time to write and visualize your impact story.
"Data storytelling is a structured method for communicating insights from data. It involves a combination of three core elements: data, visuals, and a narrative"
The presentation of your findings in charts and graphs is just the beginning of your impact story. An impact story is based on your analysis and interpretation of your data. In your story, you should not only clearly articulate what you have learned but also why it is important and what you will do with those lessons. This is a creative process where you need to consider with whom you share this information and what you want to achieve.
This also applies to data visualization. It’s not just about presenting the data but also about interpreting what the charts and graphs show and why that is important. Don’t forget that the less positive results are just as important in your impact story, just like the plot twist in a traditional story.
Use worksheet 5 for help with how to:
- Sketch the starting point using your objectives and impact goal
- Introduce your audience as characters in your story
- Outline the plot based on the main findings
- Objectively introduce your data through plot twists
- Reflect on what it all means.
Next steps
In the preparation of this module, we asked you what you wanted to achieve with an impact analysis or evaluation. Did you want to improve your digital offering for your audience? Did you want to communicate your impact more effectively to your funders for more sustainable funding? Did you want to increase the budget for digital culture at the school you collaborate with? Now you know what you want, and it’s time to turn these intentions into actions.
Even more impact
Use your insights and experiences in a future activity. Make impact thinking a part of everything you do. Curious about how others do this? Read the case studies of DutchCulture and Festival Cement.
