Posted on

August Spark News: Getting Unstuck – Equity, Advocacy, and Collective Impact

Spark Policy Institute

Are We Getting Anywhere?

Spark Policy InstituteAt Spark, we’re experts at developing actionable strategies to achieve meaningful, measurable outcomes. But in today’s complex environment, it’s sometimes challenging for our partners to see the progress they’ve made. In our August newsletter, we’re sharing resources you can apply in real-life settings to measure your progress and take positive steps forward, no matter where you are in the process of making meaningful social change happen. We’re also excited to share new efforts in understanding Collective Impact and how it is, or isn’t moving the needle on systems change.

Want to receive more updates like this? You can also subscribe to our newsletter to receive monthly updates.

Posted on

Evaluating Multi-stakeholder Advocacy Efforts

This is the second in a series of blogs on topics we’ll be presenting on at the American Evaluation Association’s (AEA) annual meeting, which will be in Atlanta, GA October 24-29.

Today’s advocacy environment is complex, with multiple stakeholders working together in campaigns that range from informal networks to collaborative impact and other similarly coordinated efforts. As a result, evaluating these initiatives is equally as complex, looking not only at outcomes, but the roles and contributions of multiple stakeholders. While advocacy evaluation has evolved over the past 10 years, transitioning from an emergent area to an established field of practice, effectively addressing the complexity of multi-stakeholder efforts that may or may not directly align remains one of the most challenging.

You can aggregate to tell the story of a group of organizations, but it’s the aggregate of individual organization evaluations, not an evaluation of a field of organizations. Rather, there is a need to understand the dynamics of how organizations – a term that may also encompass partners in government, private sector, service delivery, etc. – interact, in concert or, sometimes, even at odds. These dynamics are the key understanding how multi-stakeholder advocacy gets to impact along with understanding how organizations come together to influence policy change, build cohesive fields of practice, and accomplish more than any one group can do.

Adding to the Toolbox

This week, I will be presenting on this topic at the American Evaluation Association’s annual meeting with Jewlya Lynn here at Spark, Jared Raynor of TCC Group, and Anne Gienapp from ORS Impact. The session will look at examples of how evaluators work in multi-stakeholder environments to design different methods for collecting and analyzing data. No different from any other field of evaluation, advocacy and multi-stakeholder advocacy evaluations draw on surveys, interviews, focus groups, and observations. While these traditional methods are important, our session will take a look at other frameworks and types of analysis can help strengthen these more traditional processes, such as:

  • Multi-stakeholder advocacy toolboxAssessing mature and emergent advocacy fields, using an advocacy field framework, can help evaluators understand how a field of advocacy organizations collectively influences a specific policy area. The five dimensions of advocacy fields – field frame, skills and resources, adaptive capacity, connectivity, and composition – make it easier to untangle the concept of a field.
  • Machine learning, a data analysis approach using algorithms to generate patterns or predictions, is useful in surfacing themes in large, unstructured data sets. It can help address questions such as perceptions regarding a particular issue, differences in perceptions based on geography or language, how sentiment has changed over time, the likelihood sentiment turns to action, and how actions reflect policy decisions.
  • Dashboard tracking can help facilitate agreement on measures and create a tracking system to collect relevant data across multiple stakeholders, which is often one of the largest logistical issues faced by multi-stakeholder evaluations, particularly when the groups are working autonomously or across a wide range of activities.

Interested in learning more? Join us at our presentation: Advocacy as a Team Game: Methods for Evaluating Multi-Stakeholder Advocacy Efforts this Thursday, October 27!

Posted on

June Spark News: Changing the World, One System at a Time

Spark Policy Institute

Spark-notext-highresThis month, we’re looking at how organizations can support large-scale systems change, either as a backbone, partner, evaluator, fiscal intermediary, or through many other roles. But we would be remiss if we didn’t take a moment to talk about what happened in Orlando. Earlier in June, we witnessed the worst mass shooting in our country’s modern history. In the wake of the shooting, there has been a lot of discussion about how we got here and where we go.

As some of you may know, Spark was originally conceived to replicate, improve on, and expand the types of systems change work that one of the founders helped to lead in response to the Columbine High School shooting. During that process, over a hundred leaders from across the system, community and private sector came together to try to find a systemic solution. They found some small changes, but it took years before anything significant shifted. Spark was created to help catalyze, accelerate, learn from, and scale systems change efforts across issues and needs. It was born of a recognition that meaningful change doesn’t happen in a vacuum – it requires a cross-system, cross-sector approach.

The why of what happened at Pulse on June 12 is complex and there is no easy – or singular – way to prevent similar incidents happening in the future. But we can work toward achieving a solution together by recognizing the complexity of the situation and the ways in which we all play a part in creating, implementing, and continuing to improve that solution.

Read the rest of the newsletter. Want to receive more updates like this? You can subscribe to our newsletter here.

Posted on

The Case for Developmental Evaluation

This blog is co-authored by Marci Parkhurst and Hallie Preskill from FSG, Dr. Jewlya Lynn from Spark Policy Institute, and Marah Moore from i2i Institute. It is also posted on FSG’s website: www.fsg.org 

In a recent blog post discussing the importance of good evidence in supporting systems change work, evaluation expert Lisbeth Schorr wrote, “To get better results in this complex world, we must be willing to shake the intuition that certainty should be our highest priority…” Rather, she argues, “it is time for all of us to think more expansively about evidence as we strive to understand the world of today and to improve the world of tomorrow.” [Emphasis added]

At the annual American Evaluation Association Conference (AEA) in November, practitioners, funders, and academics from around the world gave presentations and facilitated discussions around a type of evaluation that is specifically designed to meet this need for a more expanded view of evidence. It’s called developmental evaluation, and, as noted by other commentators, it took this year’s AEA conference by storm.

What is developmental evaluation?

Developmental evaluation (DE) “is grounded in systems thinking and supports innovation by collecting and analyzing real-time data in ways that lead to informed and ongoing decision making as part of the design, development, and implementation process.” As such, DE is particularly well-suited for innovations in which the path to success is not clear. By focusing on understanding what’s happening as a new approach is implemented, DE can help answer questions such as:

  • What is emerging as the innovation takes shape?
  • What do initial results reveal about expected progress?
  • What variations in effects are we seeing?
  • How have different values, perspectives, and relationships influenced the innovation and its outcomes?
  • How is the larger system or environment responding to the innovation?

DE can provide stakeholders with a deep understanding of context and real-time insights about how a new initiative, program, or innovation should be adapted in response to changing circumstances and what is being learned along the way.

A well-executed DE will effectively balance accountability with learning; rigor with flexibility and timely information; reflection and dialogue with decision-making and action; and the need for a fixed budget with the need for responsiveness and flexibility. DE also strives to balance expectations about who is expected to adapt and change based on the information provided (i.e., funders and/or grantees).

The case for developmental evaluation

Developmental evaluation (DE) has the potential to serve as an indispensable strategic learning tool for the growing number of funders and practitioners that are focusing their efforts on facilitating systems change. But, DE is different from other approaches to evaluation. Articulating what exactly DE looks like in practice, what results it can produce, and how those results can add value to a given initiative, program, or innovation is a critical challenge, even for leaders who embrace DE in concept.

To help meet the need for a clear and compelling description of how DE differs from formative and summative evaluation and what value it can add to an organization or innovation, we hosted a think tank session at AEA in which we invited attendees to share their thoughts on these questions. We identified 4 overarching value propositions of DE, which are supported by quotes from participants:

1) DE focuses on understanding an innovation in context, and explores how both the innovation and its context evolve and interact over time.

  • “DE allows evaluators AND program implementers to adapt to changing contexts and respond to real events that can and should impact the direction of the work”.
  • “DE provides a systematic way to scan and understand the critical systems and contextual elements that influence an innovation’s road to outcomes.”
  • “DE allows for fluidity and flexibility in decision-making as the issue being addressed continues to evolve.”

2) DE is specifically designed to improve innovation. By engaging early and deeply in an exploration of what a new innovation is and how it responds to its context, DE enables stakeholders to document and learn from their experiments.

  • “DE is perfect for those times when you have the resources, knowledge, and commitment to dedicate to an innovation, but the unknowns are many and having the significant impact you want will require learning along the way.”
  • “DE is a tool that facilitates “failing smart” and adapting to emergent conditions.”

3) DE supports timely decision-making in a way that monitoring and later-stage evaluation cannot. By providing real-time feedback to initiative participants, managers, and funders, DE supports rapid strategic adjustments and quick course corrections that are critical to success under conditions of complexity.

  • “DE allows for faster decision-making with ongoing information.”
  • “DE provides real time insights that can save an innovation from wasting valuable funds on theories or assumptions that are incorrect.”
  • “DE promotes rapid, adaptive learning at a deep level so that an innovation has greatest potential to achieve social impact.”

4) Well-executed DE uses an inclusive, participatory approach that helps build relationships and increase learning capacity while boosting performance.

  • “DE encourages frequent stakeholder engagement in accessing data and using it to inform decision-making, therefore maximizing both individual and organizational learning and capacity-building. This leads to better outcomes.”
  • “DE increases trust between stakeholders or participants and evaluators by making the evaluator a ‘critical friend’ to the work.”
  • “DE can help concretely inform a specific innovation, as well as help to transform an organization’s orientation toward continuous learning.”

Additionally, one participant offered a succinct summary of how DE is different from other types of evaluation: “DE helps you keep your focus on driving meaningful change and figuring out what’s needed to make that happen—not on deploying a predefined strategy or measuring a set of predefined outcomes.”

We hope that these messages and talking points will prove helpful to funders and practitioners seeking to better understand why DE is such an innovative and powerful approach to evaluation.

Have other ideas about DE’s value? Please share them in the comments.

Learn more about developmental evaluation:

Posted on

Redefining Rigor: Describing quality evaluation in complex, adaptive settings

This blog is co-authored by Dr. Jewlya Lynn, Spark Policy Institute, and Hallie Preskill, FSG. The blog is also posted on FSG’s website: www.fsg.org 

Traditionally, evaluation has focused on understanding whether a program is making progress against pre-determined indicators. In this context, the quality of the evaluation is often measured in part by the “rigor” of the methods and scientific inquiry. Experimental and quasi-experimental methods are highly-valued and seen as the most rigorous designs, even when they may hamper the ability of the program to adapt and be responsive to its environment.

Evaluations of complex systems-change strategies or adaptive, innovative programs cannot use this same yardstick to measure quality. An experimental design is hard to apply when a strategy’s success is not fully defined upfront and depends on being responsive to the environment. As the recognition of the need for these programs, and consequently the number of complex programs grows, so does the need for a new yardstick. In recognition of this need, we proposed a new definition of rigor at the 2015 American Evaluation Association annual conference, one that broadens the ways we think of quality in evaluation to encompass things that are critical when the target of the evaluation is complex, adaptive, and emergent.

We propose that rigor be redefined to include a balance between four criteria:

  • Quality of the Thinking: The extent to which the evaluation’s design and implementation engages in deep analysis that focuses on patterns, themes and values (drawing on systems thinking); seeks alternative explanations and interpretations; is grounded in the research literature; and looks for outliers that offer different perspectives.
  • Credibility and Legitimacy of the Claims: The extent to which the data is trustworthy, including the confidence in the findings; the transferability of findings to other contexts; the consistency and repeatability of the findings; and the extent to which the findings are shaped by respondents, rather than evaluator bias, motivation, or interests.
  • Cultural Responsiveness and Context: The extent to which the evaluation questions, methods, and analysis respect and reflect the stakeholders’ values and context, their definitions of success, their experiences and perceptions, and their insights about what is happening.
  • Quality and Value of the Learning Process: The extent to which the learning process engages the people who most need the information, in a way that allows for reflection, dialogue, testing assumptions, and asking new questions, directly contributing to making decisions that help improve the process and outcomes.

The concept of balancing the four criteria is at the heart of this redefinition of rigor. Regardless of its other positive attributes, an evaluation of a complex, adaptive program that fails to take into account systems thinking will not be responsive to the needs of that program. Similarly, an evaluation that fails to provide timely information for making decisions, lacks rigor even if the quality of the thinking and legitimacy of the claims is high.

The implications of this redefinition are many.

  • From an evaluator’s point of view, it provides a new checklist of considerations when designing and implementing an evaluation. It suggests that specific, up front work will be needed to understand the cultural context, the potential users of the evaluation and the decisions they need to make, and the level of complexity in the environment and the program itself. At the same time, it maintains the same focus the traditional definition of rigor has always had on leveraging learnings from previous research and seeking consistent and repeatable findings. Ultimately, it asks the evaluator to balance the desire for the highest-quality methods and design with the need for the evaluation to have value for the end-user, and for it to be contextually appropriate.
  • From an evaluation purchaser’s point of view, it provides criteria for considering the value of potential evaluators, evaluation plans, and reports. It can be a way of articulating up-front expectations or comparing the quality of different approaches to an evaluation.
  • From a programmatic point of view, it provides a yardstick by which evaluators can not only be measured, but by which the usefulness and value of their evaluation results can be assessed. It can help program leaders and staff have confidence in the evaluation findings or have a way of talking about what they are concerned about as they look at results.

Across evaluators, evaluation purchases and users of evaluation, this redefinition of rigor provides a new way of articulating expectations from evaluation and elevating the quality and value of the evaluations. It is our hope that this balanced approach helps evaluators, evaluation purchasers and evaluation users to share ownership over the concept of rigor and finding the right balance of the criteria for their evaluations.