Posted on

Evaluators’ Varied Roles in Collective Impact

Person wearing many hats to represent varied roles

Over the next few months, we’ll be releasing a series of blogs on topics we’ll be presenting on at the American Evaluation Association’s (AEA) annual meeting, which will be in Atlanta, GA October 24-29. You can learn more about the meeting, including how to register here.

Google “Collective Impact” and you’ll get roughly 1.8 million hits (including this blog). Although collective impact (CI) is just one path out of many, it is clear the framework has taken hold as a means to tackle complex problems through a systemic lens. By their nature, however, CI initiatives are complex and emergent. The often include a mix of policy, practice, program, and alignment strategies that engage many different organizations and stakeholders. Moreover, it is not uncommon to have a diverse array of stakeholders, including funders, in the mix.

As CI grows, many different leaders are building our understanding of how to best support the work through evaluation. One thing we have come to realize is that, as varied and complex as CI initiatives are, so are the roles of their evaluators. We can be learning partners, developers of shared measurement systems, strategy partners, or even systems partners, helping align evaluation and learning throughout the system. Because of this, our effectiveness as evaluators depends on understanding which roles are needed and when, as well as how to balance these multiple roles.

Person wearing many hats to represent varied rolesIn addition to traditional formative and summative evaluation in a CI context, an evaluator may also be a:

  1. Developmental evaluator, providing real-time learning focused on supporting innovation in a complex context;
  2. Facilitator, helping partners develop and test a collective theory of change, use data to make better decisions, or align systems across evaluations;
  3. Data collector/analyzer, helping to support problem definition, identify and map the stakeholders in the system, or vet possible solutions and understand their potential for improving outcomes;
  4. Developer of system-level measures of collective capacity and impact, as well as evaluator of process of CI, providing feedback on how to strengthen it; and/or
  5. Creator of a shared measurement system, including adapting core measures to local contexts.

This October, I have the privilege to present on this topic at the American Evaluation Association’s annual meeting with Hallie Preskill from FSG, Ayo Atterberry from the Annie E. Casey Foundation, Meg Hargreaves from Community Science, and Rebecca Ochtera here at Spark Policy. Our presentation will look at the varied roles evaluators play in the CI context. It will also look at what funders and initiatives look for from the CI evaluation teams, exploring how knowing how to navigate these varied roles can help evaluation support system change, leading to more effective evaluation activities.

Interested in learning more? Join us at our presentation: The many varied and complex roles of an evaluator in a collective impact initiative!

Posted on

Working in Fields

Yellow flowers in a field

I’ve been thinking a lot lately about how different it looks to work in a field instead of alone. And no, I don’t mean out in a field of flowers (though that sounds lovely). Rather, I’m referring to a field of organizations trying to cause the same type of change, though not necessarily in collaboration or even cooperation.

We are all part of these fields: it’s the five other organizations who submitted nearly the same proposal as you did to a local funder; the three groups who knocked on the same policymaker’s door last week, talking about the same issue; the two partners you call when a quick turnaround opportunity comes up that you can’t pull off alone. The mix of all these types of organizations comprises our field (or fields, for multi-issue, multi-area organizations).

Years of emphasis on collaboration and collective impact have made sure we all recognize that we can’t get to the big wins without partners. However, we also deal with the competing reality that collaboration is hard, time consuming, and rarely exists across all the relevant organizations. So what if we thought about our work at a field level as more than just our collaborations? What would it take to influence how a field of organizations can achieve major wins together?

It turns out some folks out there have started to think about this and, in fact, have begun to define some dimensions of fields of advocates who are trying to advance a policy or systemic issue. Within each of these dimensions, there are concrete ways advocacy organizations, funders or even evaluators can help to strengthen the field:

Framing of the issue or issues

Effective fields share a common frame or core set of values underlying their work. For example, pursuing Healthy Eating, Active Living (HEAL) policy change in order to address inequities v. to address the lack of capacity in the healthcare system to meet the growing demand. While each of these frames is valid, one would approach the problem fairly differently and identify different solutions within each frame. Because of this, they might be different fields.

  • So what can you do with this? You can look for partners who share the same frame and identify common opportunities to act together. You can promote your frame to other organizations in the same space and work to change the overall framing around the issue. As a funder, you can invest resources to strengthen organizations that share your frame.

Resources and skills

Fields are composed of organizations with different resources and skills to influence an issue. Most fields have deficits, such as a lack of strong policy analysis/research capacity or insufficient community organizing. Alternatively, they may lack key skills that are rarely needed, but when needed they are critical, such as launching ballot initiative campaigns or leading litigation processes.

  • So what can you do about this? Explore the deficits and seek to grow your organization in that direction, rather than duplicating already available capacity. Build the skills of other organizations so they can engage in work that is complimentary to your own.

Connectivity

Fields have varying levels of relationships between organizations. Strong relationships allow for coordinated strategies, leveraging of capacities, and use of common messaging on specific policy opportunities, while weak relationships can make it difficult to work together at the right moments to achieve policy or systemic changes.

  • So what can you do about this? Seek out organizations that are traditionally not connected to your part of the field, particularly those that bring a different resource, skill, or voice to the work. Intentionally leverage old and new partners for concrete opportunities to move an issue together. If you’re a funder, provide resources and convening opportunities to organizations currently not connected to one another.

Composition

Composition refers to the representation of different types of stakeholders, from the inclusion of public/private partners to racial, ethnic, and economic diversity and more. Fields that represent a broad array of stakeholders carry more influence when policy opportunities arise and also help craft policy solutions that are more likely to achieve the desired outcomes than when only a couple perspectives dominate the field.

  • So what can you do about this? Identify which voices are missing from the field or are marginalized. Expand the perspectives or organizations you engage. If you’re a funder, consider bringing new voices into the field by funding direct service or community organizations who want to advocate.

Adaptive Capacity

When the context shifts in a policy campaign or systems building strategy, effective advocacy organizations shift their strategies as well. A strong field doesn’t shift in 10 different directions or miss key signals indicating a shift is needed. Rather, when part of the field identifies the need for change, the need is recognized throughout the field and the changes are aligned.

  • So what can you do about this? If your organization is skilled at monitoring the environment, share what you’re learning actively with other organizations. If you don’t have the capacity to do that monitoring, seek out partners who do and share what you learn from them. If you’re a funder, consider funding one or more organizations to engage in environmental assessments ongoing with the expectation that they will disseminate the learning actively and in a timely manner.

This might be the longest blog I’ve ever written, but I hope you find the ideas are worth the number of words on the screen. Working at a field level may lead to stronger collaborations in the future, but just as important is the way it will change how organizations respond and react to each other and the environment in order to advocate in ways that collectively contribute to the likelihood of success.

I will be joining thought leaders on this issue of working collectively (without having to work collaboratively) at the American Evaluation Association’s annual conference this year. If you’re attending, I hope you can join us and move this dialogue forward.

Posted on

How do you know if you’re getting the best quality in your evaluations?

How do you know if you’re getting the best quality in your evaluations?

RigorQuality in evaluation used to be defined as rigor (and sometimes still is), with rigor meaning the competence of the evaluator, the legitimacy of the process and, of course, applying the best research methods to the collection and analysis of data. These are important, but they don’t count as an all-encompassing definition of quality, particularly in complex, adaptive settings where evaluation partners with strategy.

If we cannot count of these measures to define quality, what are alternative ways of understanding if your evaluation is high quality? Hallie Preskill from FSG and I will be joining forces at the American Evaluation Association’s annual conference this Friday to explore this issue. We are proposing that the concept of “rigor” (and thus what you can look for in your evaluations) can – and should – be redefined as:

  • Balancing whether the evaluation is useful, inclusive of multiple perspectives, unbiased, accurate, and timely.
  • The quality of the learning process, including whether it engages the people who need the information when they need the information.
  • The quality of the thinking, including whether the evaluation engages in deep analysis, seeks alternative explanations, situates findings within the literature, and uses systems thinking.
  • The credibility and legitimacy of the findings, including whether people are confident in the ‘truth’ being presented.
  • Responsiveness to the cultural context, including the integration of stakeholders’ values and definitions of success, as well as who helps to interpret the findings.

Capture

Are you attending the annual conference? Come join us for an interactive discussion on how to reframe rigor and quality in your evaluations.