Posted on

Collective Impact Study Update

By Jewlya Lynn, Founder and Chief Learning Officer, Spark Policy Institute; and Sarah Stachowiak, CEO, ORS Impact

Back in May, 2017 ORS Impact and Spark Policy Institute embarked on an ambitious and important study to explore how collective impact contributes to changes in systems and populations through its unique approach to addressing social issues. We are pleased to update the field on our study progress as we near the end of our data collection.

Study Selection Process

To seek out a representative group of sites that would allow us to dig into how collective impact leads to systems and population changes, we invited the field to nominate sites that were good examples of how the collective impact model is contributing to changes in systems and outcomes.

We screened over 150 sites to find examples of mature initiatives (at least three years old), in the US and Canada, with evidence of strongly implementing all of the five conditions of collective impact, changing a number of systems, and moving the needle on outcomes. Of the 39 sites that met our initial screening criteria, 25 initiatives across a wide range of topics and geographies consented to participate in our inquiry. Our research steering team helped us vet the final set of sites and identify any potential challenges.

Methods

To develop a broad understanding of how collective impact works across many different settings, we interviewed two key people in each of the 25 sites who had a deep knowledge of the initiative– often backbone leaders or steering team members. In addition, we reviewed many documents that described activities, goals, and progress. Through the interviews and documents, we sought to understand how collective impact shows up in different initiatives and how the collective impact conditions might be linked to important changes in systems and outcomes.

Drilling Down to Equity and Impact

Next, we selected two sets of sites to dive deeper into our primary research question of “what is the contribution of collective impact to population and systems changes?” and to explore more explicitly how equity intersects with the CI model– a priority identified by our steering team as an opportunity to raise up a significant principle that is important to the field.

To address our primary research question about the contribution of CI, we selected eight “contribution” sites  from our original pool of 25. Our goal in selection was NOT to identify the “best” CI sites, but rather to identify a set of initiatives that allowed the best chance of examining causal linkages among how collective impact is implemented and the changes that occur in people, organizations, systems, and ultimate impact. Therefore, we selected eight sites that had the strongest evidence that all five collective impact conditions were present and that multiple social and systems changes could  clearly be linked to population changes achieved.

Within our contribution sites, we collected additional data using structured group dialogues to help us understand how collective impact is implemented and what challenges the initiatives faced in implementation. In addition, we facilitated group process tracing sessions where a group of stakeholders pressure tested theories of change that the systems changes and population changes they were experiencing could be attributed to their collective impact efforts and not to external events or context.

To address growing interest and urgency around infusing equity into collective impact work, we also identified three sites to allow us to understand more deeply the issues related to meaningful and authentic inclusion of beneficiary communities in CI planning, implementation and leadership; what types of equity focused strategies are being implemented; what factors are related to “readiness” to engage in equity work; and how initiatives are achieving equity-focused systems changes and outcomes. ARISE– an initiative focused on the needs of indigenous students in Anchorage, Promesa Boyle Heights– a community-driven initiative in Los Angeles, and RGV Focus– a regional initiative focused on low-income Hispanic children and families in the Rio Grande valley– all provide unique opportunities to learn how CI can actively engage the families and communities they intend to benefit and how having an equity focus interacts with the CI model.

Coming up in future posts, we’ll share our collective impact and equity rubrics that we are using to understand how the model is being implemented, and blogs on systems changes and process tracing as a methodology. As we wrap up our analysis and distill findings, we’ll also provide a glimpse into our initial results and share lessons with the field from what we are learning.

Posted on

Three Tips for Making Network Analysis Actionable for Your Social Impact Project

Three Tips for making network analysis actionable for your social impact project

Joby Schaffer, MA, Associate Researcher at Spark Policy InstituteMany of our partners have adopted what Jed Miller and Rob Stuart called “Network-Centric Thinking.” They recognize that long-term sustainable progress on today’s social problems rarely comes from the efforts of a single organization. Rather, progress requires a strategy involving networks of organizations with the aim of producing network effects.

However, the strategist and evaluator’s task of connecting network strategy to network effects to final outcomes is often difficult, not least because networks are embedded in complex, adaptive systems in which cause and effect relationships are rarely straightforward. Moreover, because quantitative social network analysis (SNA) is often new to many social impact organizations, it is easy to get bogged down in superficial findings to the determinant of more actionable insights.

Three Tips for making network analysis actionable for your social impact projectThere are now a large number of resources on designing network analyses for complex evaluations (see some of our favorites below), but we’ve found three tips particularly useful for ensuring a network analysis yields actionable insights. In short, a design for evaluating a network should:

  • Start by adopting a framework for how network structure leads to network effects;
  • Avoid the lure of only using quantitative SNA; and  
  • Design your network analysis with future data collections in mind: connecting change in the network to outcomes is one of the most powerful insights you’ll uncover.  

Get a Framework

Our partners often make use of theories of change, systems maps, scenario mapping, power analyses, and other tools to frame the nature of the problem they want to address and to develop strategies to guide their work. For learning partners like us, these tools are often a key part of developing and shaping evaluation questions and hypotheses. However, because network theory is relatively new to most people, the expected impact of network strategies is often underspecified in these documents.

For example:

An initiative may agree that the presence of working relationships among cross-sector partners is an important interim outcome…

       …with the expectation these partnerships will help address an upstream driver of a problem…

       …but they may not fully consider how the strengths and weaknesses of the current network structure alters
their chances of activating this “network effect”…

      …which in turn limits their understanding of which actions are needed to advance the network strategy.

Frameworks help to address these problems because they relate network structure to network effects. For example, Peter Plastrik and Madeleine Taylor describe three networks [pdf] on the basis of the depth of their connections – connectivity, alignment, and production. If an initiative aims for cross-sector collaboration (production), but the initial network analysis reveals little connectivity between organizations, it’s best to engage in more connectivity-related and alignment-related network building tasks before encouraging project collaboration.

Choose a Multi-method Approach

When most people think about network analysis, they think of network maps or strange-sounding network statistics like density or centrality. This is quantitative SNA, and it is an essential tool for describing structural properties of a network. Among other things, an SNA will reveal gaps in the network (e.g. perhaps organizations from a certain sector are underrepresented), show areas of deep or shallow connections (e.g connectivity among one subset and alignment among another subset), and identify which organizations play important roles in the network (e.g. bring unique partners to the network).

However, if used alone, SNA may mask a lot of the network information leaders need to make effective decisions. For example, network strategy often involves developing structures for coordination, including convenings, working groups, and shared measurement systems. While it’s possible to use SNA to wrangle some insights about whether these coordinating efforts lead to more effective partnerships, it’s often more meaningful to hear from participants how these structures influenced their work. In short, interviews are much better at capturing the organizational and inter-organizational effects of the network – innovations, greater efficiencies realized, knowledge and information shared, etc.

Design with the Future in Mind

It is good practice to design any evaluation with pre- and post-interventions in mind. Especially for quantitative SNA, it is worth the upfront time to identify what you hope your network will look like in the future, not just examine it today. Repeated network maps can show how the network is evolving over time, which is a great way to identify how coordinating efforts are producing network-level effects (e.g., better representation of certain sectors at convening events, connections made between subsets of the networks, etc.). Again, adopting a framework can be very useful. Many frameworks explicitly describe the stages of network evolution and provide guidance on how to identify and manage a network in transition.

The more social change agents adopt network-centric thinking, the better the chances we’ll make real progress on today’s social problems. We can support this mindset by ensuring our network analyses produce actionable insights. We’ve found these three tips are useful to our work. Based on your experiences, what other tips do you recommend?

New to network thinking or network analysis? Here’s a few of our favorite resources.