Posted on

Redefining Rigor: Describing quality evaluation in complex, adaptive settings

This blog is co-authored by Dr. Jewlya Lynn, Spark Policy Institute, and Hallie Preskill, FSG. The blog is also posted on FSG’s website: www.fsg.org 

Traditionally, evaluation has focused on understanding whether a program is making progress against pre-determined indicators. In this context, the quality of the evaluation is often measured in part by the “rigor” of the methods and scientific inquiry. Experimental and quasi-experimental methods are highly-valued and seen as the most rigorous designs, even when they may hamper the ability of the program to adapt and be responsive to its environment.

Evaluations of complex systems-change strategies or adaptive, innovative programs cannot use this same yardstick to measure quality. An experimental design is hard to apply when a strategy’s success is not fully defined upfront and depends on being responsive to the environment. As the recognition of the need for these programs, and consequently the number of complex programs grows, so does the need for a new yardstick. In recognition of this need, we proposed a new definition of rigor at the 2015 American Evaluation Association annual conference, one that broadens the ways we think of quality in evaluation to encompass things that are critical when the target of the evaluation is complex, adaptive, and emergent.

We propose that rigor be redefined to include a balance between four criteria:

  • Quality of the Thinking: The extent to which the evaluation’s design and implementation engages in deep analysis that focuses on patterns, themes and values (drawing on systems thinking); seeks alternative explanations and interpretations; is grounded in the research literature; and looks for outliers that offer different perspectives.
  • Credibility and Legitimacy of the Claims: The extent to which the data is trustworthy, including the confidence in the findings; the transferability of findings to other contexts; the consistency and repeatability of the findings; and the extent to which the findings are shaped by respondents, rather than evaluator bias, motivation, or interests.
  • Cultural Responsiveness and Context: The extent to which the evaluation questions, methods, and analysis respect and reflect the stakeholders’ values and context, their definitions of success, their experiences and perceptions, and their insights about what is happening.
  • Quality and Value of the Learning Process: The extent to which the learning process engages the people who most need the information, in a way that allows for reflection, dialogue, testing assumptions, and asking new questions, directly contributing to making decisions that help improve the process and outcomes.

The concept of balancing the four criteria is at the heart of this redefinition of rigor. Regardless of its other positive attributes, an evaluation of a complex, adaptive program that fails to take into account systems thinking will not be responsive to the needs of that program. Similarly, an evaluation that fails to provide timely information for making decisions, lacks rigor even if the quality of the thinking and legitimacy of the claims is high.

The implications of this redefinition are many.

  • From an evaluator’s point of view, it provides a new checklist of considerations when designing and implementing an evaluation. It suggests that specific, up front work will be needed to understand the cultural context, the potential users of the evaluation and the decisions they need to make, and the level of complexity in the environment and the program itself. At the same time, it maintains the same focus the traditional definition of rigor has always had on leveraging learnings from previous research and seeking consistent and repeatable findings. Ultimately, it asks the evaluator to balance the desire for the highest-quality methods and design with the need for the evaluation to have value for the end-user, and for it to be contextually appropriate.
  • From an evaluation purchaser’s point of view, it provides criteria for considering the value of potential evaluators, evaluation plans, and reports. It can be a way of articulating up-front expectations or comparing the quality of different approaches to an evaluation.
  • From a programmatic point of view, it provides a yardstick by which evaluators can not only be measured, but by which the usefulness and value of their evaluation results can be assessed. It can help program leaders and staff have confidence in the evaluation findings or have a way of talking about what they are concerned about as they look at results.

Across evaluators, evaluation purchases and users of evaluation, this redefinition of rigor provides a new way of articulating expectations from evaluation and elevating the quality and value of the evaluations. It is our hope that this balanced approach helps evaluators, evaluation purchasers and evaluation users to share ownership over the concept of rigor and finding the right balance of the criteria for their evaluations.

Posted on

How do you know if you’re getting the best quality in your evaluations?

How do you know if you’re getting the best quality in your evaluations?

RigorQuality in evaluation used to be defined as rigor (and sometimes still is), with rigor meaning the competence of the evaluator, the legitimacy of the process and, of course, applying the best research methods to the collection and analysis of data. These are important, but they don’t count as an all-encompassing definition of quality, particularly in complex, adaptive settings where evaluation partners with strategy.

If we cannot count of these measures to define quality, what are alternative ways of understanding if your evaluation is high quality? Hallie Preskill from FSG and I will be joining forces at the American Evaluation Association’s annual conference this Friday to explore this issue. We are proposing that the concept of “rigor” (and thus what you can look for in your evaluations) can – and should – be redefined as:

  • Balancing whether the evaluation is useful, inclusive of multiple perspectives, unbiased, accurate, and timely.
  • The quality of the learning process, including whether it engages the people who need the information when they need the information.
  • The quality of the thinking, including whether the evaluation engages in deep analysis, seeks alternative explanations, situates findings within the literature, and uses systems thinking.
  • The credibility and legitimacy of the findings, including whether people are confident in the ‘truth’ being presented.
  • Responsiveness to the cultural context, including the integration of stakeholders’ values and definitions of success, as well as who helps to interpret the findings.

Capture

Are you attending the annual conference? Come join us for an interactive discussion on how to reframe rigor and quality in your evaluations.

Posted on

A Better Place: Contemplating systems change in place based work

What does it take to trigger systemic changes that can improve quality of life in a community? One suggestion that emerged from the Better Place convening in San Diego earlier this month was the idea of novelty: we need a novel idea and then opportunities to test it, discover its value, and take it to scale.

But where do novel ideas come from? And how do you know which novel idea is worth trying – will be the right one to solve the problems you’re trying to solve, to get to the outcomes you’re trying to achieve?

Better Place 1

Novel ideas might emerge from listening to and working with communities. Not only can community members describe what does and doesn’t work in their communities, they can help you diagnose the problem in ways that surface new opportunities. But this doesn’t happen from a focus group or a one-time community meeting. It takes sustained, meaningful partnerships to move from hearing about the impact of the system on communities to surfacing novel ways of solving the problems.

Novel ideas might also emerge from outside the community, or from community members who have had experiences in other places. Sometimes community members and systems leaders are all so embedded in the current systems it can be hard to see where change is possible. Having an outside approach, another city’s successes, a new framework brought to the table by a credible participant in the process can trigger a novel idea.

Novel ideas might emerge from failures as well. A systems change attempt that did not get adopted might trigger thinking about what could be adopted. Alternatively, a systems change that was adopted, but did not achieve the desired outcomes, might inspire new thinking about what it will take to get to the outcomes.

These are all things evaluation can help with: creating space to work with, and learn from, communities; bringing in insights from other places; and learning from failure. Evaluation can also be a partner in assessing the potential of a novel idea to solve the problem.

Better Place 2Not every systems change idea is worth adopting; in fact most ideas are probably not worth pursuing. But how can you tell when your novel idea is the right idea – when it can meaningfully improve outcomes for your community in the ways you care about?

  • You can look at the data and make sure that the novel idea directly addresses the things the data tell you are the drivers of the problem. Of course, data can’t tell you the solution that will address those drivers, unless someone is already piloting a similar solution in your community.
  • You can look at what has happened in other communities around the country, looking for similar examples in similar contexts to see how it played out.
  • You can work with community members to vet options and explore scenarios of what will happen if the policy is adopted and how the community will look different.

Or, ideally, you can do all three together, using data and examples from other communities to inform a community dialogue around the options and generate scenarios of what will happen if the policy is adopted. Once again, evaluation can be a powerful partner in all three of these steps to designing the best systemic changes.

Getting to a better place is not something anyone can do alone, but one of my biggest takeaways from the Better Place convening was the many stakeholders who believe in the potential of evaluators to be partners at the table, helping to shape the change by bringing data and evaluation to the table as decision-making tools. Each partner in a systems change effort has to contribute in a meaningful way and evaluators have contributions to make as partners in getting to meaningful changes, not neutral judges of what worked and what didn’t.

Are you an evaluator who is trying to shift from a traditional evaluation role to being a partner at the table? Check out our Developmental Evaluation Toolkit for some tips and processes to make it easier to adopt the new role.

Posted on

Evaluating Collaboration in Place-based Initiatives: Can it Move the Needle?

On October 5th and 6th, I will have the opportunity to facilitate a session on how evaluation can help stakeholders understand and strengthen cross-sector partnerships and collaboration more broadly at the Art & Science of Place-Based Evaluation. The conference is hosted by Jacobs Center for Neighborhood Innovation, the Aspen Institute Forum for Community Solutions, and the Neighborhood Funders Group and builds off of a series of on-going national conversations about the importance of “place” in philanthropic investments, including the Center of Philanthropy and Public Policy’s National Forum on Place-based Initiatives and the Aspen Institutes’ Promising Practices Conference.

If you Google “evaluate collaboration” you will see there is no shortage of tools for assessing the strength of a collaborative effort, but as I prepared for the session, I found myself asking: Is the quality of collaboration really the most important thing to investigate with your limited evaluation resources?

Effectively engaging partners in place-based work depends on more than good processes and practices. Among other things, it requires:

  • Meaningfully engaging different sectors to leverage the different motivations bringing each to the table (which requires surfacing and understanding those motivations!);
  • Tackling difficult power dynamics, sometimes evident in the room, but other times they play out in how strategies are implemented:
  • Recognizing and responding appropriately to the impact of the cultural assumptions participants bring to the process;
  • Managing the negative consequences of failed attempts to work collaboratively in the past;
  • Effectively leveraging large networks of organizations and leaders, often larger than the initiative has time to meaningfully engage and manage; and
  • Engaging with communities experiencing disparities in ways that are appropriate and lead to an impact on the work.

In addition, there is the fundamental issue of whether and how the structures and processes of collaboration are leading to something worthwhile – moving the needle on the issue that brought everyone together. Are collaboration and engagement managed in ways that advance the work or only in ways that advance the quality of collaboration?

If evaluation is going to play an role in helping place-based initiatives advance their collaboration processes, and get to the meaningful change, it needs to go beyond tools and become a real-time partner in uncovering motivations, power dynamics, and cultural assumptions; it needs to help pick apart how networks are functioning and where engagement might be most effective; and it should play a role in understanding how, and to what extent, nontraditional partners are influencing the decisions being made and contributing to shifts in the overall strategy and direction of the work.

These are the types of issues we’ll be exploring in the collaboration and cross-sector partnerships session at the convening. Don’t worry, you’ll leave with a list of evaluation tools that can be helpful if you want to focus on evaluating the effectiveness of your collaborative processes. But you’ll also leave with insights about how to engage evaluation in helping you tackle the fundamental issues standing between good collaboration and having an impact on the issues that matter.

Interested in learning more about the conference or attending? Visit the conference website: http://www.jacobscenter.org/placebased/

Want to hear from more facilitators?  Check out the blog from Meg Long of Equal Measure about connecting community change to systems change and Sonia Taddy-Sandino of Harder+Company about “getting ready” for place-based work. Interested in accessing new resources before the conference?  Check out our toolkits on engaging nontraditional voices and decision-making in complex, multi-stakeholder settings.

Posted on

Using Learning To Do Good, Even Better

One of the best parts of my job is helping organizations use learning to do good, even better. Recently, we worked with Project Health Colorado, a strategy funded by The Colorado Trust with support from The Colorado Health Foundation, focused on building public will to achieve access to health for all Coloradans by fostering a statewide discussion about health care and how it can be improved. The strategy included fourteen grantees and a communications campaign working independently and together to build public will. It also combined an impact evaluation with coaching on engaging in real-time, data-driven strategic learning to help grantees and The Trust test and adapt their strategies to improve outcomes.

Lessons learned about strategic learning:

So, how can organizations use real time learning to tackle a complex strategy in a complex environment – building will around a highly politicized issue? Our strategic learning model built the capacity of The Trust and grantees to engage in systematic data collection, along with collective interpretation and use of information to improve strategies. As a result, grantees shifted strategies in real time, increasing their ability to influence audience awareness of access to health issues and willingness to take action.

As a result of the learning, The Trust made major changes to the overarching strategy including shifting from asking grantees to use a prepackaged message to using the “intent” of the message with training on how to adapt it. This was particularly important for grantees working with predominately minority communities, who reported the original message did not resonate in their communities.

The real-time learning was effective because it allowed grantees and the Trust to practice interpreting and using the results of systematic data collection, applying what they learned to improve their strategies. The evaluation also supported adaptation over accountability to pre-defined plans, creating a culture of adaptation and helping participants strategize how to be effective at building will.

Lessons learned about evaluation:

The evaluation focused learning at the portfolio level, looking at the collective impact on public will across all grantee strategies. As the evaluator charged with figuring out the impact of this strategy, where everyone was encouraged to constantly adapt and improve, we learned that having multiple in-depth data collection methods, tailored to the ways different audiences engaged in the strategy, and explicitly planning for how to capture emergent outcomes allowed the evaluation to stay relevant even as the strategy shifted.

Rad Resources:

Want to learn more?

This post originally appeared September 14, 2015 on AEA365, the American Evaluation Association blog. The American Evaluation Association is an international professional association of evaluators devoted to the application and exploration of program evaluation, personnel evaluation, technology, and many other forms of evaluation. The American Evaluation Association is celebrating Nonprofits and Foundations Topical Interest Group (NPFTIG) Week. The contributions all this week to aea365 come from our NPFTIG members.