Posted on

Tips to Ensuring Your Evaluation is a Learning Tool

It used to be that evaluation was something that happened occasionally, was sometimes required by funders, and could sometimes be useful. Not anymore. When you ask a room full of people who are trying to make a meaningful difference whether they have been part of an evaluation, almost everyone raises their hand. But what that means – being part of an evaluation – differs greatly from person to person.

Evaluation can be about accountability, meeting a funder requirement, or even providing information back to your board. At its best, though, evaluation is about learning – learning what worked, what didn’t, why it didn’t, what to do better next time.

 

Focusing your evaluation on learning

The earlier you decide to use your evaluation as a learning process, the more likely it is that it will work out that way. As your evaluator is planning the evaluation design, talk to them about what you want to know, what will help you to do good, even better. Try exploring some of the questions below with your evaluator – each of these question may bubble up an important area for learning!

  • Are we implementing a strategy that is new to us, something we’ve never tried before?
  • Is there a strategy we’re implementing in a new way or with a new target population?
  • Is there a strategy we’ve been implementing for years, but never assessed to see if it really makes a difference?
  • Is there information about our external environment that will help us to tailor our strategies and better achieve our outcomes?
  • Is there information about our own organization’s internal capacity that will help us know if we’re ready to implement a strategy?

If these questions do not help you narrow your learning, try filling in the blank on this statement instead:

  • Unless we know ____________________________, we won’t know how to improve our strategy.

Focus your evaluation on both Strategies and Outcomes

This may sound obvious to some of you, but it bears repeating. When we want to learn from our evaluation, we need to understand the relationship between the strategy we are implementing and the outcomes we are achieving. Let’s use some definitions:

Strategy: A defined approach to implementing multiple smaller actions (tactics), which are collectively intended to achieve a particular outcome. Strategies are completely within your control to implement or not implement as you choose.

 

Outcome: The change you hope to influence as a result of your strategies. It may be a change in people (yourself, your target audience, community members) or even a change in the environment (built environment, health of habitats, etc.). You cannot control whether it occurs, only do your best to influence the change.

Work with your evaluator to make sure that they are capturing the relationship between decisions and actions that are within your control and the outcomes they achieve, those things you can only hope to influence. When your evaluator presents results, telling you whether or not you’ve achieved your outcomes, nothing is worse than not being able to figure out what to change to improve your outcomes.

 

Intuitive learning matters too!

We’ll talk more about this in another blog coming soon, but here’s the teaser: Your evaluator, no matter how amazing, does not have the intuitive and on-the-ground knowledge of the people implementing the program. Understanding what an evaluation is really telling you means taking into account your intuitive knowledge of what is happening in your program. To learn from an evaluation, you need high quality evaluation results, but you also need to explore what the evaluation results mean with the people who understand your program best, including staff, volunteers, and even consumers.

Posted on

This is going to take us 10 years! What do we tell our funders now?

Many things worth doing cannot be done quickly. Changing public policy systems, building communities, tackling complex social issues, and advocating for meaningful change are not things one undertakes and completes in a single year, three years, or sometimes even in ten years.

Yet, at the same time, we all have funders and our funders need to know that they are funding organizations that are making a meaningful difference – not ten years from now, but today. Not to mention that it would also be helpful to our long-term efforts if we could figure out whether today’s activities are making a difference for tomorrow.

This is where strategic roadmaps and interim outcomes come in. Yes, the terms are full of jargon, but they are worth learning because they can help you and your funder understand whether what you are doing really does matter and help you do good, even better.

Is what we’re doing today going to matter tomorrow?

Strategic Roadmaps are a powerful tool for taking a complex problem and breaking down the solution into a series of meaningful, smaller changes on the way to the big success. They  are similar to Theories of Change, but provide a higher level of detail and focus on interim outcomes. At Spark, we use them for evaluation (where the concept first came from), but we’ve also adapted them to be a strategic planning tool.

The first thing you figure out in a roadmapping process is where you’re going to end up. It’s a lot like a vision statement, only you want to be very concrete and realistic. Don’t say that you’re going to eliminate obesity. Say  you will increase the number of residents who meet recommendations for physical activity, fruit and veggie consumption, and caloric intake (this is straight from the Stapleton Foundation’s be well initiative roadmap). Or, instead of saying you’ll cause all schools to purchase all of their food locally through farm to school programs, say you’ll cause collaborative, sustainable implementation of farm to school statewide (this one is from the Colorado Farm to School Task Force’s roadmap).

Once you know where you are going to end up, you map it backwards – asking yourself step by step what has to be in place to get to where you want to go. That’s how you know whether what you’re doing today will directly contribute to achieving the end of your road – because you can visually show the road you’re on and how one thing steadily leads to another until lasting and meaningful change is achieved.

For more information about what a Strategic Roadmap process looks like, visit our blog on How Groups Become Change Agents.

How do we know if what we’re doing is getting us to where we want to go?

Here is where the concept of interim outcomes comes in. When you know where you want to end up, and you know what has to happen each step of the way to get there, you’ve just defined your interim outcomes. Let’s back out of that jargon for a minute though and instead just talk about two things: control and influence.

The things you do every day – the meetings you host, the number of people you screen, the trainings you convene, the legislators you talk to – these are things within your control. Sometimes we get into the habit of just reporting to our stakeholders and funders a laundry list of things we did, things in our control. We might tell people we talked to 10 legislators, recruited 50 people to attend our meeting, or screened 200 residents. Careful though, those aren’t outcomes! They are just counts that tell you what you’ve been up to.

Then there are all the things we can’t control, but that really matter to us – the things we are constantly trying to influence. They are things like how a legislator votes on a specific bill, what the 50 people at our meeting think about the issue when they walk out the door and the actions they commit to taking, or the number of people we screened who followed up with their doctors. We work hard to cause these things to happen, but at the end of the day, we can only hope to influence them. We aren’t inside people’s minds, we can’t force these changes and actions to occur. This is what makes them our outcomes.

So what is an interim outcome? It is an outcome that is along your road. You need it to happen in order to get a little closer to that big, long-term goal. But it isn’t something you can control. Are you trying to increase the number of residents meet recommendations for physical activity, fruit and veggie consumption, and caloric intake? Your interim outcomes might include completion rates of residents attending your programs, increases in physical activity level, or increases in knowledge about healthy eating and active living. These are things that matter, but they don’t matter in and of themselves, they matter because they are supposed to be leading to the end of your road.

Strategic roadmaps and interim outcomes sound great, but how do I use these with my funders?

If you know your end of the road and your interim outcomes, you are in a great position to sit down with your funders and talk to them about why your work really matters. You can also work with them to agree on which interim outcomes are achievable within their funding period and could be part of an evaluation. When you’re having this conversation, don’t forget to include capacity building as an important outcome they should support and evaluation should capture. If you are trying to do something that will take three, five, or ten years, you need to build the organizational capacity to do the work for the long haul. Here are some quick tips for talking with your funder about long-term change:

  • Bring a visual of your roadmap and walk through it with them in person. They may challenge you on assumptions you’ve made about why one thing will lead to another, but they will respect the effort you took to map out how you are getting from where you are today to a big and meaningful change.
  • Circle the outcomes on your visual that you think can be achieved during the funding cycle. Talk to the funder about why those are realistic in the timeframe, but other interim outcomes are not.
  • Focus the conversation on the outcomes you’ll achieve, not the activities you’ll use. Anyone can implement a community meeting or host a training. Stand out by showing the funder you are an organization who really knows how to cause change in an individual or a community.
  • Don’t go into the conversation alone. Instead, bring your community leaders, volunteers, advocates, or organizational partners to reinforce their belief that the road you are on will make a difference. We don’t try to cause meaningful changes alone, so there is no reason to be by ourselves when we attempt to secure the funding we need to make a difference.