Posted on

Embracing Values in Evaluation Practice

Research has traditionally defined rigor as obtaining an unbiased estimate of impact, suggesting the need for experimental or quasi-experimental methods and objective, quantitative measures in order to obtain trustworthy results.

I’ve spent the past few months as a member of Colorado’s Equitable Evaluation Collaboratory, which aims to examine the role evaluation plays in supporting or inhibiting progress toward equity and identifying opportunities to integrate equitable evaluation principles into practice. In particular, I’ve reflected on how the research tradition has impacted evaluation’s working orthodoxies including the notion that “credible evidence comes from quantitative data and experimental research” and “evaluators are objective.”

On the surface, these statements don’t appear particularly problematic, but dig a little deeper and we begin to see how value judgments are an integral part of how we practice evaluation. The types of projects we take on, the questions we ask, the frameworks we use, the types of data we collect, and the ways we interpret results – are all deeply rooted in what we value. As an evaluator focused on use, I aim to make these practice decisions in partnership with my clients; however, suggesting that I, or any evaluator, does not play an active role in making these decisions discounts our inherent position of power.

Now that I’ve tuned into the orthodoxies, I see them everywhere, often dominating the conversation. In a meeting last week, a decision-maker was describing the path forward for making a controversial policy decision. He wanted to remove subjectivity and values from the conversation by developing guidelines rooted in “evidence-based practice” and turned to me to present the “facts.”

As a proponent of data-driven decision making, I value the role of evidence; however, there is a lot to unpack behind what we have declared – through traditional notions of rigor – “works” to improve health and social outcomes. Looking retrospectively at the evidence, and thinking prospectively about generating new knowledge, it’s time to ask ourselves some hard questions, including:

  • What interventions do we choose to study? Who developed them? Why did they develop them?
  • What have we (as a society) chosen not to investigate?
  • What population have we “tested” our interventions on? Have we looked for potentially differential impacts?
  • What outcomes do we examine? Who identified these impacts to be important?
  • Who reported the outcomes? Whose perspective do we value?
  • What time-period do we examine? Is that time-period meaningful to the target population?
  • Do we look for potentially unintended consequences?

As we begin to unpack the notion of “what works” we begin to see the decision-points, the values and the inherent power and privilege in what it means to be an evaluator. It is time that we owned the notion that what we choose to study and how we choose to measure success are not objective, rather, they are inherently subjective. And importantly, our choices communicate values.

So how do we begin to embrace our role? As a step forward, I have started including a discussion of values, both mine and my clients, at the beginning of a project and clarifying how those values will influence the evaluation scope and process. Explicitly naming the importance of equity during the evaluative process has helped keep the goals of social change and social justice front and center.  Naming values helps stakeholders acknowledge their power and provides a lens through which to make decisions.

Equitable evaluation is an expedition into the unknown, requiring a transformation in how we conceptualize our role as evaluator. Having taken my initial steps into the Upside Down, I look forward to the many unknowns.

In what way do you see values showing up in your evaluative work?

 

Posted on

The Collective Impact Research Study: What is all this really going to mean, anyway?

By Jewlya Lynn, CEO, Spark Policy Institute; Sarah Stachowiak, CEO, ORS Impact

It’s easy for evaluators to sometimes get tied up in the technical terms around our work, leaving lay people unclear on what some of our decisions and choices mean.  Without care, we can also risk being opaque about what a particular design can and can’t do.  With this blog, we want to untangle what we think our design will tell us, and what it won’t do.

With this research study, ORS Impact and Spark Policy Institute are seeking to understand the degree to which the collective impact approach contributed meaningfully to observed positive changes in people’s’ lives (or, in some cases, species or ecosystems).  In other words, when and under what conditions did collective impact make a difference where we’re seeing positive changes, or are there other explanations or more significant contributors to identified changes?  While we’ll learn a lot more than just that, at its heart, that’s what this study will do.  

Our primary approach to understand the core question around contribution and causal relationships will be to use process tracing.  Process tracing provides a rigorous and structured way to identify and explore competing explanations for why change happens and to determine the necessity and sufficiency of different kinds of evidence to support different explanations that we’ll find through our data collection efforts.

To implement the process tracing, we will dig deeply into data around successful changes—a population change or set of changes plausibly linked to the CI efforts—within six sites.  We’ll explore these changes and their contributing factors with data from existing documents, interviews with site informants, focus groups with engaged individuals, and a participatory process to review and engage in sense-making with stakeholders around the ways in which we understand change to have happened.  We’ll try and untangle the links between implementation of the collective impact approach and early outcomes, the links between early outcomes and systems changes, and the links between systems changes and ultimate impacts.

Figure:  Diagram of “Process” for Tracing

Note:  Future blogs will provide more information on the different rubrics we’ve developed and are using.

Using a process tracing approach also means that we’ll explicitly explore alternate hypotheses for why change happened—was there another more impactful initiative?  Was there a federal funding stream that supported important related work?  Was there state policy that paved the way that was unconnected to stakeholders’ work?  Would these changes have occurred whether collective impact was around or not?

Additionally, we’ll look at two sites where we would expect to see change but don’t, with the expectation that these sites can help us understand if the patterns we’re seeing at successful sites are absent or showing up differently, findings that would help give us more confidence that the patterns we’re seeing are meaningful.

Process tracing as our approach does mean that our unit of analysis—the sphere within which we will be exploring change and causal relationships—is going to be approximately eight sites.  While we hope to find sites where a cluster of impact outcomes result from a specific set of activities (or “process”), we are choosing to go deeply in a few sites with an approach that will provide rigor around how we develop and confirm our understanding of the relationships between activities and changes.  And because we are looking across diverse sites, working on varied issue areas (e.g., food systems, education, environmental issues, etc.) and at different scales (e.g., cities, multiple counties, entire states), identifying patterns across diverse contexts will increase our confidence around what collective impact conditions, principles and other contextual factors are most related to these successes.

With more data around if and when we find causal relationships, we will also go back to our data set of 22 sites that we are also engaging with early to see if we can, likewise, find similar patterns to those found through the process tracings.  For these sites, we’ll use data we will have collected on their fidelity to collective impact, efforts around equity, successes with different types of systems changes, and types of ultimate impacts.  Are we seeing similar patterns around the necessity of fidelity to certain conditions?  Are we seeing similar patterns in the relationship between certain types of systems changes and impacts?

Despite the strengths we believe this study has, it will not be the end-all-be-all, final say on the efficacy of collective impact.  All studies have limitations, and we want to be clear about those as well.  Given time and resources, we can’t conduct in-depth evaluations of the full range of efforts and activities any given collective impact site is undertaking.  Our unit of analysis isn’t a full site; it won’t take in the full complexity of the history of the initiative, or the full array of activities and efforts.  For example, it’s likely that a site that we engage with around a particular success has also experienced areas with no discernable progress.  We also are not comparing collective impact to other change models.  That doesn’t make the exploration of causality around successful changes less meaningful, but it does mean that we’ll understand contribution to specific changes well rather than understanding and judging the success of collective impact at a community-level or comparing collective impact to other models of driving systemic change.

We do believe that this study will fill a gap in the growing body of research, evaluation and evidence around collective impact by deeply understanding contribution in particular cases and by looking at a diverse and varied set of cases.  The social sector will benefit from continued interrogation of collective impact using many methods, units of analysis and approaches.  In the end, the more we learn, the better we can make meaningful progress on the gnarly issues that face vulnerable places and populations.

Posted on

Sharing as We Go: The Collective Impact Research Study

By Jewlya Lynn, CEO, Spark Policy Institute; Sarah Stachowiak, CEO, ORS Impact

 

 

At Spark Policy Institute (Spark) and ORS Impact (ORS), we have been doing systems building work for over a decade. When the Collective Impact approach came along, it created a level of clarity for many people, both about what it means to change systems as well as providing clear insights about how to do so.

And now, six years into the CI approach creating momentum and excitement, many systems change leaders find themselves asking the questions:

“Does the Collective Impact (CI) approach directly contribute to systems changes that leads to population changes?  When does it contribute and in what ways?  And most importantly, what does that mean for our work?”

We at ORS Impact and Spark Policy Institute are excited to have the opportunity to answer the first two questions in partnership with 30 collective impact sites in the US and Canada as part of the Collective Impact Research Study. Our goal is to provide the type of information that can help with the third question – putting the learning into action.

Research and evaluation of CI, particularly when testing the efficacy of the approach, must be approached differently than program evaluation or a more-straightforward descriptive study. It is not sufficient to collect self-report data about activities and changes occurring in the system, even with a verification process, without having a clear understanding of the types of changes that matter and the types of impact desired.

Consequently, our approach will consider how the external environment and CI initiatives have evolved over time and support an understanding of the causal relationship between CI efforts and their outcomes.  As part of the study, we will seek to understand the range of ways CI is experienced and perceived, the implications of these differences on its effectiveness, and the implications for how the approach is deployed and supported.

Together, Spark and ORS bring extensive expertise in the study of complex initiatives. We know communities, organizations, and funders, and we know what it means to fully participate in a long-term initiative that involves multiple individuals, organizations, and systems moving toward a common goal of change. We also bring a healthy skepticism about the approach and how the five conditions and principles come together to drive systemic change.

We are also acutely aware of the need for a credible, actionable study. We will be following rigorous research practices and providing a high level of transparency around our methods.  To that end, we want to share some high-level attributes of our study and lay out some of the content we will be providing along the way.

Research Study Phases

ORS and Spark are approaching this research in a multiphase process that will allow us to use multiple methods that will add rigor and enhance our ability to make useful comparisons across disparate sites while focusing on answering the primary causal question.  Our research will occur through three phases:

  • Develop a set of analytic rubrics that will provide the foundation for all our research activities. These analytic rubrics will be grounded in the conditions and principles of CI, as well as approaches for tracking systems changes, equity and population-level changes.
  • Examine extant data, review documents, and collect new high-level data across a broad set of ~30 CI initiatives to understand more broadly how CI initiatives are implementing the conditions and principles of the approach and their systems change outcomes and population-level impacts. As you may have seen in outreach from the CI Forum, we used an open nomination process to help ensure our sample for this stage is broad and diverse in its initiative issue areas, origins, and funding sources.
  • Dive more deeply in a focused group of 8 CI initiatives initially evaluated as part of the first phase of site analysis to better understand the conditions that support or impede population success. Our goal in this phase is to examine the implementation of the CI approach and more deeply understand the degree to which different causal explanations can be supported in different contexts and with differing levels of success in achieving population outcomes. We are using a method called process tracing, which is a qualitative analysis approach that helps understand causal inferences by interrogating rival hypotheses to explain changes observed (we will describe process tracing in detail in a future blog post).

Future Blog Topics

To continue in our efforts to bring transparency to this work, we will be blogging each month about this study, presenting our methods and specific rubrics we will be using as well as providing examples and lessons learned. Please check back each month for blogs on the following topics.

  • Early June: Design details and list of sites being included in the study.
  • June and July: Three-part series discussing the rubrics being used for this study: CI, systems change, and equity.
  • August: A description of process tracing and an example.
  • September: Key lessons from untangling cause and effect of CI and population outcomes.
  • October: A case study example from one site evaluated.
  • November/December: Key findings from the study.
  • January: Final report release via the CI Forum.

We encourage you to share any of your insights about CI in the comments section below!

Posted on

Evaluators’ Varied Roles in Collective Impact

Person wearing many hats to represent varied roles

Over the next few months, we’ll be releasing a series of blogs on topics we’ll be presenting on at the American Evaluation Association’s (AEA) annual meeting, which will be in Atlanta, GA October 24-29. You can learn more about the meeting, including how to register here.

Google “Collective Impact” and you’ll get roughly 1.8 million hits (including this blog). Although collective impact (CI) is just one path out of many, it is clear the framework has taken hold as a means to tackle complex problems through a systemic lens. By their nature, however, CI initiatives are complex and emergent. The often include a mix of policy, practice, program, and alignment strategies that engage many different organizations and stakeholders. Moreover, it is not uncommon to have a diverse array of stakeholders, including funders, in the mix.

As CI grows, many different leaders are building our understanding of how to best support the work through evaluation. One thing we have come to realize is that, as varied and complex as CI initiatives are, so are the roles of their evaluators. We can be learning partners, developers of shared measurement systems, strategy partners, or even systems partners, helping align evaluation and learning throughout the system. Because of this, our effectiveness as evaluators depends on understanding which roles are needed and when, as well as how to balance these multiple roles.

Person wearing many hats to represent varied rolesIn addition to traditional formative and summative evaluation in a CI context, an evaluator may also be a:

  1. Developmental evaluator, providing real-time learning focused on supporting innovation in a complex context;
  2. Facilitator, helping partners develop and test a collective theory of change, use data to make better decisions, or align systems across evaluations;
  3. Data collector/analyzer, helping to support problem definition, identify and map the stakeholders in the system, or vet possible solutions and understand their potential for improving outcomes;
  4. Developer of system-level measures of collective capacity and impact, as well as evaluator of process of CI, providing feedback on how to strengthen it; and/or
  5. Creator of a shared measurement system, including adapting core measures to local contexts.

This October, I have the privilege to present on this topic at the American Evaluation Association’s annual meeting with Hallie Preskill from FSG, Ayo Atterberry from the Annie E. Casey Foundation, Meg Hargreaves from Community Science, and Rebecca Ochtera here at Spark Policy. Our presentation will look at the varied roles evaluators play in the CI context. It will also look at what funders and initiatives look for from the CI evaluation teams, exploring how knowing how to navigate these varied roles can help evaluation support system change, leading to more effective evaluation activities.

Interested in learning more? Join us at our presentation: The many varied and complex roles of an evaluator in a collective impact initiative!

Posted on

The Case for Developmental Evaluation

This blog is co-authored by Marci Parkhurst and Hallie Preskill from FSG, Dr. Jewlya Lynn from Spark Policy Institute, and Marah Moore from i2i Institute. It is also posted on FSG’s website: www.fsg.org 

In a recent blog post discussing the importance of good evidence in supporting systems change work, evaluation expert Lisbeth Schorr wrote, “To get better results in this complex world, we must be willing to shake the intuition that certainty should be our highest priority…” Rather, she argues, “it is time for all of us to think more expansively about evidence as we strive to understand the world of today and to improve the world of tomorrow.” [Emphasis added]

At the annual American Evaluation Association Conference (AEA) in November, practitioners, funders, and academics from around the world gave presentations and facilitated discussions around a type of evaluation that is specifically designed to meet this need for a more expanded view of evidence. It’s called developmental evaluation, and, as noted by other commentators, it took this year’s AEA conference by storm.

What is developmental evaluation?

Developmental evaluation (DE) “is grounded in systems thinking and supports innovation by collecting and analyzing real-time data in ways that lead to informed and ongoing decision making as part of the design, development, and implementation process.” As such, DE is particularly well-suited for innovations in which the path to success is not clear. By focusing on understanding what’s happening as a new approach is implemented, DE can help answer questions such as:

  • What is emerging as the innovation takes shape?
  • What do initial results reveal about expected progress?
  • What variations in effects are we seeing?
  • How have different values, perspectives, and relationships influenced the innovation and its outcomes?
  • How is the larger system or environment responding to the innovation?

DE can provide stakeholders with a deep understanding of context and real-time insights about how a new initiative, program, or innovation should be adapted in response to changing circumstances and what is being learned along the way.

A well-executed DE will effectively balance accountability with learning; rigor with flexibility and timely information; reflection and dialogue with decision-making and action; and the need for a fixed budget with the need for responsiveness and flexibility. DE also strives to balance expectations about who is expected to adapt and change based on the information provided (i.e., funders and/or grantees).

The case for developmental evaluation

Developmental evaluation (DE) has the potential to serve as an indispensable strategic learning tool for the growing number of funders and practitioners that are focusing their efforts on facilitating systems change. But, DE is different from other approaches to evaluation. Articulating what exactly DE looks like in practice, what results it can produce, and how those results can add value to a given initiative, program, or innovation is a critical challenge, even for leaders who embrace DE in concept.

To help meet the need for a clear and compelling description of how DE differs from formative and summative evaluation and what value it can add to an organization or innovation, we hosted a think tank session at AEA in which we invited attendees to share their thoughts on these questions. We identified 4 overarching value propositions of DE, which are supported by quotes from participants:

1) DE focuses on understanding an innovation in context, and explores how both the innovation and its context evolve and interact over time.

  • “DE allows evaluators AND program implementers to adapt to changing contexts and respond to real events that can and should impact the direction of the work”.
  • “DE provides a systematic way to scan and understand the critical systems and contextual elements that influence an innovation’s road to outcomes.”
  • “DE allows for fluidity and flexibility in decision-making as the issue being addressed continues to evolve.”

2) DE is specifically designed to improve innovation. By engaging early and deeply in an exploration of what a new innovation is and how it responds to its context, DE enables stakeholders to document and learn from their experiments.

  • “DE is perfect for those times when you have the resources, knowledge, and commitment to dedicate to an innovation, but the unknowns are many and having the significant impact you want will require learning along the way.”
  • “DE is a tool that facilitates “failing smart” and adapting to emergent conditions.”

3) DE supports timely decision-making in a way that monitoring and later-stage evaluation cannot. By providing real-time feedback to initiative participants, managers, and funders, DE supports rapid strategic adjustments and quick course corrections that are critical to success under conditions of complexity.

  • “DE allows for faster decision-making with ongoing information.”
  • “DE provides real time insights that can save an innovation from wasting valuable funds on theories or assumptions that are incorrect.”
  • “DE promotes rapid, adaptive learning at a deep level so that an innovation has greatest potential to achieve social impact.”

4) Well-executed DE uses an inclusive, participatory approach that helps build relationships and increase learning capacity while boosting performance.

  • “DE encourages frequent stakeholder engagement in accessing data and using it to inform decision-making, therefore maximizing both individual and organizational learning and capacity-building. This leads to better outcomes.”
  • “DE increases trust between stakeholders or participants and evaluators by making the evaluator a ‘critical friend’ to the work.”
  • “DE can help concretely inform a specific innovation, as well as help to transform an organization’s orientation toward continuous learning.”

Additionally, one participant offered a succinct summary of how DE is different from other types of evaluation: “DE helps you keep your focus on driving meaningful change and figuring out what’s needed to make that happen—not on deploying a predefined strategy or measuring a set of predefined outcomes.”

We hope that these messages and talking points will prove helpful to funders and practitioners seeking to better understand why DE is such an innovative and powerful approach to evaluation.

Have other ideas about DE’s value? Please share them in the comments.

Learn more about developmental evaluation: