Posted on

The Case for Developmental Evaluation

This blog is co-authored by Marci Parkhurst and Hallie Preskill from FSG, Dr. Jewlya Lynn from Spark Policy Institute, and Marah Moore from i2i Institute. It is also posted on FSG’s website: www.fsg.org 

In a recent blog post discussing the importance of good evidence in supporting systems change work, evaluation expert Lisbeth Schorr wrote, “To get better results in this complex world, we must be willing to shake the intuition that certainty should be our highest priority…” Rather, she argues, “it is time for all of us to think more expansively about evidence as we strive to understand the world of today and to improve the world of tomorrow.” [Emphasis added]

At the annual American Evaluation Association Conference (AEA) in November, practitioners, funders, and academics from around the world gave presentations and facilitated discussions around a type of evaluation that is specifically designed to meet this need for a more expanded view of evidence. It’s called developmental evaluation, and, as noted by other commentators, it took this year’s AEA conference by storm.

What is developmental evaluation?

Developmental evaluation (DE) “is grounded in systems thinking and supports innovation by collecting and analyzing real-time data in ways that lead to informed and ongoing decision making as part of the design, development, and implementation process.” As such, DE is particularly well-suited for innovations in which the path to success is not clear. By focusing on understanding what’s happening as a new approach is implemented, DE can help answer questions such as:

  • What is emerging as the innovation takes shape?
  • What do initial results reveal about expected progress?
  • What variations in effects are we seeing?
  • How have different values, perspectives, and relationships influenced the innovation and its outcomes?
  • How is the larger system or environment responding to the innovation?

DE can provide stakeholders with a deep understanding of context and real-time insights about how a new initiative, program, or innovation should be adapted in response to changing circumstances and what is being learned along the way.

A well-executed DE will effectively balance accountability with learning; rigor with flexibility and timely information; reflection and dialogue with decision-making and action; and the need for a fixed budget with the need for responsiveness and flexibility. DE also strives to balance expectations about who is expected to adapt and change based on the information provided (i.e., funders and/or grantees).

The case for developmental evaluation

Developmental evaluation (DE) has the potential to serve as an indispensable strategic learning tool for the growing number of funders and practitioners that are focusing their efforts on facilitating systems change. But, DE is different from other approaches to evaluation. Articulating what exactly DE looks like in practice, what results it can produce, and how those results can add value to a given initiative, program, or innovation is a critical challenge, even for leaders who embrace DE in concept.

To help meet the need for a clear and compelling description of how DE differs from formative and summative evaluation and what value it can add to an organization or innovation, we hosted a think tank session at AEA in which we invited attendees to share their thoughts on these questions. We identified 4 overarching value propositions of DE, which are supported by quotes from participants:

1) DE focuses on understanding an innovation in context, and explores how both the innovation and its context evolve and interact over time.

  • “DE allows evaluators AND program implementers to adapt to changing contexts and respond to real events that can and should impact the direction of the work”.
  • “DE provides a systematic way to scan and understand the critical systems and contextual elements that influence an innovation’s road to outcomes.”
  • “DE allows for fluidity and flexibility in decision-making as the issue being addressed continues to evolve.”

2) DE is specifically designed to improve innovation. By engaging early and deeply in an exploration of what a new innovation is and how it responds to its context, DE enables stakeholders to document and learn from their experiments.

  • “DE is perfect for those times when you have the resources, knowledge, and commitment to dedicate to an innovation, but the unknowns are many and having the significant impact you want will require learning along the way.”
  • “DE is a tool that facilitates “failing smart” and adapting to emergent conditions.”

3) DE supports timely decision-making in a way that monitoring and later-stage evaluation cannot. By providing real-time feedback to initiative participants, managers, and funders, DE supports rapid strategic adjustments and quick course corrections that are critical to success under conditions of complexity.

  • “DE allows for faster decision-making with ongoing information.”
  • “DE provides real time insights that can save an innovation from wasting valuable funds on theories or assumptions that are incorrect.”
  • “DE promotes rapid, adaptive learning at a deep level so that an innovation has greatest potential to achieve social impact.”

4) Well-executed DE uses an inclusive, participatory approach that helps build relationships and increase learning capacity while boosting performance.

  • “DE encourages frequent stakeholder engagement in accessing data and using it to inform decision-making, therefore maximizing both individual and organizational learning and capacity-building. This leads to better outcomes.”
  • “DE increases trust between stakeholders or participants and evaluators by making the evaluator a ‘critical friend’ to the work.”
  • “DE can help concretely inform a specific innovation, as well as help to transform an organization’s orientation toward continuous learning.”

Additionally, one participant offered a succinct summary of how DE is different from other types of evaluation: “DE helps you keep your focus on driving meaningful change and figuring out what’s needed to make that happen—not on deploying a predefined strategy or measuring a set of predefined outcomes.”

We hope that these messages and talking points will prove helpful to funders and practitioners seeking to better understand why DE is such an innovative and powerful approach to evaluation.

Have other ideas about DE’s value? Please share them in the comments.

Learn more about developmental evaluation:

Posted on

Planning for Adaptation

Logo 8I’ve spent a lot of time over the last decade thinking about, experimenting with, and refining tools for planning in complex, adaptive settings. As we put together Spark’s Adaptive Planning Toolkit, I’ve had the opportunity to reflect back and think about the genesis of the tools and what we have learned over the years.

I have tremendous admiration for all of the partners I’ve worked with who have tackled complex problems with adaptive approaches. That they can work amid such great uncertainty is impressive in and of itself, but the fact that they are willing to approach solving the problems in ways that are, themselves, uncertain and untested is even more laudable.

The stakeholders who came together to prevent another tragedy like the Columbine school shooting not only didn’t know how to integrate the many different service systems to prevent a future shooting, they were also brand new to systems mapping, which was a critical part of developing a plan for change. I remember the walls covered with boxes and lines, as participants tried to break down how the system functioned today in order to figure out how it could function tomorrow.

DLPLogoFINALThe leaders who formed the core of the Daylight Project, focused on improving access to behavioral health services for deaf and hard of hearing consumers, similarly tackled a complex problem using tools that were untested and new to them. Consumer stories helped inform their work along the way, but so did real-time strategic learning, which included gathering data about their environment and forecasting the likelihood of success for each partner organization they invited to join the effort.

Scenario MappingRecently, The Colorado Health Foundation used an adaptive planning process to develop their Consumer Advocacy funding strategy. Using scenario planning tools, mapping of current funding, and even a post-mortem, they went all out with adaptive planning. Unlike the previous examples, by this point Spark, as their partners in crime, had a well-established repertoire of adaptive planning tools. However, similar to the experiences in the first two examples, this approach was still new and out of the comfort zone for the organization, yet they embraced it fully and developed a truly creative, results focused, and adaptive funding strategy.

I am personally very excited to share our adaptive planning tools. I believe in them. I have seen them help many different types of groups make a meaningful difference on truly difficult problems. I also believe this idea of adaptive planning is a work in progress – we have some pieces pulled together, but by no means is this the be all, end all of planning in complex settings. I am excited to learn how others are doing adaptive planning and hope you will participate by sharing your stories and building our common base of tools for how to do this difficult work.

Posted on

Spark’s New Year’s Resolutions to Support Sustainable Change

Spark Policy InstituteDon’t you just love thinking up New Year’s resolutions? I know I do – they’re so full of the rich promise of that makes Change, capital C, so alluring. A fresh start – an opportunity to draw a line in the sand and say, “On the other side of this line, things will be different!” No matter how difficult to solve, no matter how complex the problem, making and owning a resolution is, literally, to renew your resolve to solve that complex problem. That in itself is an important step towards meaningful and sustainable change. And that’s why we at Spark are making a few resolutions for 2013 – to remind ourselves why we’re here with you on the front lines of the battle to create change in our communities.

 

Six Resolutions for Making a Meaningful Difference

So here are Spark’s Resolutions for 2013 . . .

Spark Policy Institute

  1. We will embrace meaningful change even when it’s hard for us.
    • We try to do this every day, but it isn’t easy! So we enforce upon ourselves regular pauses for reflection. We ask ourselves “Is this strategy moving us towards the change we desire?” “Is that change meaningful?” “What can we do to make sure we’re on the right track?”

 

  1. We will use data to guide our decisions, along with intuition and experience.Spark Policy Institute
    • For example, we’re planning some changes to our monthly webinar series in 2013, based on your feedback. More expert advice, more real-world examples to illustrate our points, and more interactivity. You spoke, and we listened.

 

  1.  We will engage all stakeholders, including those most affected by policies and services. Critique helps us do good, even better!
    • So please keep that feedback coming – informed change is better than arbitrary change! Comment on our blogs, attend and critique our monthly webinars, email us to tell us what you think of the resources in our Igniting Change website, and most of all, tell us how we’re doing in our projects with you – morning, noon, and night.
  1. We will keep learning, even when it means giving up our old ways of doing things.Spark Policy Institute
    • We’re all about learning, so we engage in regular quality assurance check-ins on our projects. And we’re always refining our quality assurance methods so that we can help create high-quality, customized solutions to our clients’ problems.
  1. We will work on the things we love, because passion gets results.
    • Our clients get the best outcomes when we have passion for the topic. We’re always focused on outcomes, but in 2013 we’re making sure to bring the passions of our team to bear on every project.
  1. We will be healthy! Research says dark chocolate is good for you, so we’re going for it!Spark Policy Institute

 

Spark Policy Institute

Got any New Year’s Resolutions of your own? We’d love to hear about them! And best wishes for a sparkling new year from all of us at Spark!

Posted on

Introducing Spark Policy

At Spark Policy Institute, we are dedicated to helping communities and policymakers solve complex problems.  The Spark blog will focus on concrete strategies and actions you can take as you seek to make a meaningful difference on issues that are challenging, complicated, and critically important to you and your community.

 

What are complex problems?

Called “intractable problems” by some, complex problems have a mix of stakeholders at all levels of government, each of whom have different funding sources, mandates, and expectations; these problems also have private stakeholders, consumers, and communities that cannot be left out.

What are some examples?

  • As regards the obesity epidemic, finding the policy and individual behavior change solutions that can reduce obesity and its associated illnesses;
  • With healthcare access and reform, moving beyond playing politics and into making a difference;
  • With water policy in the arid west, finding solutions that balance environmental, agricultural, and population needs;
  • Healthy food access and its intersection with transportation infrastructure, water, land-use planning, nutrition education, and schools; and
  • Behavioral health and its intersection with all aspects of our lives, from workplace productivity to juvenile justice involvement to physical health outcomes.

Spark’s work is dedicated to the challenge of addressing complex problems such as these, bringing together a combination of research, engagement of all stakeholders, and information dissemination to help find solutions.

 

What do we know about these complex problems?

We recognize that regardless of which policy arena a problem emerges from, common issues are often present:

  • Policy solutions identified in one arena are likely to cause unintended consequences in others;
  • Money talks – part of identifying any policy solution is understanding how public and private funding operates, what the limitations are, and where to find opportunities to leverage;
  • Sustainable solutions and change in the status-quo are only successful when a wide range of stakeholders are involved in identifying both the problem and the solution; and
  • Finding solutions is only the beginning – implementing change is a long, slow process that requires commitment, resources, regular evaluation and feedback, and engagement of all the stakeholders.

 

How do we solve these complex problems?

As Spark has grown, we have built skills and expertise to tackle complex problems in a wide variety of arenas: human services, health, behavioral health, natural resources, agriculture, housing, juvenile justice, criminal justice, education, early childhood, and diversity / disparities.

The Spark team we have assembled over the years now includes a mixture of:

  • Researchers who are adept at working in messy, complex settings and bring a wide variety of methodologies to their work including fiscal and legal research, evaluation, network analysis, q-methodology, focus groups, and many other quantitative and qualitative approaches;
  • Facilitators who understand how to inform dialogue with external information and input, and can create a safe environment where all stakeholders, including community members, consumers, and even youth, can participate fully in complex policy dialogues;
  • Project managers whose approach reflects the needs of their clients, and who can remain flexible as the policy environment changes; and
  • Product developers, who specialize in ensuring reports, white papers, presentations, and other materials are rich in information and attractive in presentation, but more importantly, are committed to making sure no product becomes yet another report that sits on a shelf.

 

What does a “solution” look like?

Every problem has a different solution, and we know that the solutions that are first tried often fail to address fully all the complexity of the problem.  What does a solution look like?  There is no single answer – every system is different.  Maybe the solution includes changes in how funding is utilized by government agencies.  Maybe it includes changes to policies related to access to care.  The solution might be about how non-profits mobilize and educate their communities.  It may also include new voices having a say in the decision-making process.  Sometimes a solution is about changing how existing policies are implemented and sometimes it requires an overhaul of laws and regulations.

 

Join Us

Join the Spark Team in our dedication to solving complex problems.  What are the issues facing your community?  How can you tackle them?  Each week, the Spark blog will release new tips, tools, research, and information to help you find those solutions.

Do you have any questions you want answered?  Please let us know the topics you want to learn more about!