Posted on

Evaluators’ Varied Roles in Collective Impact

Person wearing many hats to represent varied roles

Over the next few months, we’ll be releasing a series of blogs on topics we’ll be presenting on at the American Evaluation Association’s (AEA) annual meeting, which will be in Atlanta, GA October 24-29. You can learn more about the meeting, including how to register here.

Google “Collective Impact” and you’ll get roughly 1.8 million hits (including this blog). Although collective impact (CI) is just one path out of many, it is clear the framework has taken hold as a means to tackle complex problems through a systemic lens. By their nature, however, CI initiatives are complex and emergent. The often include a mix of policy, practice, program, and alignment strategies that engage many different organizations and stakeholders. Moreover, it is not uncommon to have a diverse array of stakeholders, including funders, in the mix.

As CI grows, many different leaders are building our understanding of how to best support the work through evaluation. One thing we have come to realize is that, as varied and complex as CI initiatives are, so are the roles of their evaluators. We can be learning partners, developers of shared measurement systems, strategy partners, or even systems partners, helping align evaluation and learning throughout the system. Because of this, our effectiveness as evaluators depends on understanding which roles are needed and when, as well as how to balance these multiple roles.

Person wearing many hats to represent varied rolesIn addition to traditional formative and summative evaluation in a CI context, an evaluator may also be a:

  1. Developmental evaluator, providing real-time learning focused on supporting innovation in a complex context;
  2. Facilitator, helping partners develop and test a collective theory of change, use data to make better decisions, or align systems across evaluations;
  3. Data collector/analyzer, helping to support problem definition, identify and map the stakeholders in the system, or vet possible solutions and understand their potential for improving outcomes;
  4. Developer of system-level measures of collective capacity and impact, as well as evaluator of process of CI, providing feedback on how to strengthen it; and/or
  5. Creator of a shared measurement system, including adapting core measures to local contexts.

This October, I have the privilege to present on this topic at the American Evaluation Association’s annual meeting with Hallie Preskill from FSG, Ayo Atterberry from the Annie E. Casey Foundation, Meg Hargreaves from Community Science, and Rebecca Ochtera here at Spark Policy. Our presentation will look at the varied roles evaluators play in the CI context. It will also look at what funders and initiatives look for from the CI evaluation teams, exploring how knowing how to navigate these varied roles can help evaluation support system change, leading to more effective evaluation activities.

Interested in learning more? Join us at our presentation: The many varied and complex roles of an evaluator in a collective impact initiative!

Data Use and Shared Accountability

This is the first of four briefs developed as part of The Colorado Trust’s Early Childhood Council Health Integration Evaluation. The evaluation focus was on building a more integrated system of care at the local level by linking child health services to other early childhood services. This brief examines data use and shared accountability, which grantees identified early on as a topic relevant to systems-building work.

Posted on

Redefining Rigor: Describing quality evaluation in complex, adaptive settings

This blog is co-authored by Dr. Jewlya Lynn, Spark Policy Institute, and Hallie Preskill, FSG. The blog is also posted on FSG’s website: www.fsg.org 

Traditionally, evaluation has focused on understanding whether a program is making progress against pre-determined indicators. In this context, the quality of the evaluation is often measured in part by the “rigor” of the methods and scientific inquiry. Experimental and quasi-experimental methods are highly-valued and seen as the most rigorous designs, even when they may hamper the ability of the program to adapt and be responsive to its environment.

Evaluations of complex systems-change strategies or adaptive, innovative programs cannot use this same yardstick to measure quality. An experimental design is hard to apply when a strategy’s success is not fully defined upfront and depends on being responsive to the environment. As the recognition of the need for these programs, and consequently the number of complex programs grows, so does the need for a new yardstick. In recognition of this need, we proposed a new definition of rigor at the 2015 American Evaluation Association annual conference, one that broadens the ways we think of quality in evaluation to encompass things that are critical when the target of the evaluation is complex, adaptive, and emergent.

We propose that rigor be redefined to include a balance between four criteria:

  • Quality of the Thinking: The extent to which the evaluation’s design and implementation engages in deep analysis that focuses on patterns, themes and values (drawing on systems thinking); seeks alternative explanations and interpretations; is grounded in the research literature; and looks for outliers that offer different perspectives.
  • Credibility and Legitimacy of the Claims: The extent to which the data is trustworthy, including the confidence in the findings; the transferability of findings to other contexts; the consistency and repeatability of the findings; and the extent to which the findings are shaped by respondents, rather than evaluator bias, motivation, or interests.
  • Cultural Responsiveness and Context: The extent to which the evaluation questions, methods, and analysis respect and reflect the stakeholders’ values and context, their definitions of success, their experiences and perceptions, and their insights about what is happening.
  • Quality and Value of the Learning Process: The extent to which the learning process engages the people who most need the information, in a way that allows for reflection, dialogue, testing assumptions, and asking new questions, directly contributing to making decisions that help improve the process and outcomes.

The concept of balancing the four criteria is at the heart of this redefinition of rigor. Regardless of its other positive attributes, an evaluation of a complex, adaptive program that fails to take into account systems thinking will not be responsive to the needs of that program. Similarly, an evaluation that fails to provide timely information for making decisions, lacks rigor even if the quality of the thinking and legitimacy of the claims is high.

The implications of this redefinition are many.

  • From an evaluator’s point of view, it provides a new checklist of considerations when designing and implementing an evaluation. It suggests that specific, up front work will be needed to understand the cultural context, the potential users of the evaluation and the decisions they need to make, and the level of complexity in the environment and the program itself. At the same time, it maintains the same focus the traditional definition of rigor has always had on leveraging learnings from previous research and seeking consistent and repeatable findings. Ultimately, it asks the evaluator to balance the desire for the highest-quality methods and design with the need for the evaluation to have value for the end-user, and for it to be contextually appropriate.
  • From an evaluation purchaser’s point of view, it provides criteria for considering the value of potential evaluators, evaluation plans, and reports. It can be a way of articulating up-front expectations or comparing the quality of different approaches to an evaluation.
  • From a programmatic point of view, it provides a yardstick by which evaluators can not only be measured, but by which the usefulness and value of their evaluation results can be assessed. It can help program leaders and staff have confidence in the evaluation findings or have a way of talking about what they are concerned about as they look at results.

Across evaluators, evaluation purchases and users of evaluation, this redefinition of rigor provides a new way of articulating expectations from evaluation and elevating the quality and value of the evaluations. It is our hope that this balanced approach helps evaluators, evaluation purchasers and evaluation users to share ownership over the concept of rigor and finding the right balance of the criteria for their evaluations.

Posted on

How do you know if you’re getting the best quality in your evaluations?

How do you know if you’re getting the best quality in your evaluations?

RigorQuality in evaluation used to be defined as rigor (and sometimes still is), with rigor meaning the competence of the evaluator, the legitimacy of the process and, of course, applying the best research methods to the collection and analysis of data. These are important, but they don’t count as an all-encompassing definition of quality, particularly in complex, adaptive settings where evaluation partners with strategy.

If we cannot count of these measures to define quality, what are alternative ways of understanding if your evaluation is high quality? Hallie Preskill from FSG and I will be joining forces at the American Evaluation Association’s annual conference this Friday to explore this issue. We are proposing that the concept of “rigor” (and thus what you can look for in your evaluations) can – and should – be redefined as:

  • Balancing whether the evaluation is useful, inclusive of multiple perspectives, unbiased, accurate, and timely.
  • The quality of the learning process, including whether it engages the people who need the information when they need the information.
  • The quality of the thinking, including whether the evaluation engages in deep analysis, seeks alternative explanations, situates findings within the literature, and uses systems thinking.
  • The credibility and legitimacy of the findings, including whether people are confident in the ‘truth’ being presented.
  • Responsiveness to the cultural context, including the integration of stakeholders’ values and definitions of success, as well as who helps to interpret the findings.

Capture

Are you attending the annual conference? Come join us for an interactive discussion on how to reframe rigor and quality in your evaluations.

Posted on

A Better Place: Contemplating systems change in place based work

What does it take to trigger systemic changes that can improve quality of life in a community? One suggestion that emerged from the Better Place convening in San Diego earlier this month was the idea of novelty: we need a novel idea and then opportunities to test it, discover its value, and take it to scale.

But where do novel ideas come from? And how do you know which novel idea is worth trying – will be the right one to solve the problems you’re trying to solve, to get to the outcomes you’re trying to achieve?

Better Place 1

Novel ideas might emerge from listening to and working with communities. Not only can community members describe what does and doesn’t work in their communities, they can help you diagnose the problem in ways that surface new opportunities. But this doesn’t happen from a focus group or a one-time community meeting. It takes sustained, meaningful partnerships to move from hearing about the impact of the system on communities to surfacing novel ways of solving the problems.

Novel ideas might also emerge from outside the community, or from community members who have had experiences in other places. Sometimes community members and systems leaders are all so embedded in the current systems it can be hard to see where change is possible. Having an outside approach, another city’s successes, a new framework brought to the table by a credible participant in the process can trigger a novel idea.

Novel ideas might emerge from failures as well. A systems change attempt that did not get adopted might trigger thinking about what could be adopted. Alternatively, a systems change that was adopted, but did not achieve the desired outcomes, might inspire new thinking about what it will take to get to the outcomes.

These are all things evaluation can help with: creating space to work with, and learn from, communities; bringing in insights from other places; and learning from failure. Evaluation can also be a partner in assessing the potential of a novel idea to solve the problem.

Better Place 2Not every systems change idea is worth adopting; in fact most ideas are probably not worth pursuing. But how can you tell when your novel idea is the right idea – when it can meaningfully improve outcomes for your community in the ways you care about?

  • You can look at the data and make sure that the novel idea directly addresses the things the data tell you are the drivers of the problem. Of course, data can’t tell you the solution that will address those drivers, unless someone is already piloting a similar solution in your community.
  • You can look at what has happened in other communities around the country, looking for similar examples in similar contexts to see how it played out.
  • You can work with community members to vet options and explore scenarios of what will happen if the policy is adopted and how the community will look different.

Or, ideally, you can do all three together, using data and examples from other communities to inform a community dialogue around the options and generate scenarios of what will happen if the policy is adopted. Once again, evaluation can be a powerful partner in all three of these steps to designing the best systemic changes.

Getting to a better place is not something anyone can do alone, but one of my biggest takeaways from the Better Place convening was the many stakeholders who believe in the potential of evaluators to be partners at the table, helping to shape the change by bringing data and evaluation to the table as decision-making tools. Each partner in a systems change effort has to contribute in a meaningful way and evaluators have contributions to make as partners in getting to meaningful changes, not neutral judges of what worked and what didn’t.

Are you an evaluator who is trying to shift from a traditional evaluation role to being a partner at the table? Check out our Developmental Evaluation Toolkit for some tips and processes to make it easier to adopt the new role.