Posted on

The Collective Impact Research Study: What is all this really going to mean, anyway?

By Jewlya Lynn, CEO, Spark Policy Institute; Sarah Stachowiak, CEO, ORS Impact

It’s easy for evaluators to sometimes get tied up in the technical terms around our work, leaving lay people unclear on what some of our decisions and choices mean.  Without care, we can also risk being opaque about what a particular design can and can’t do.  With this blog, we want to untangle what we think our design will tell us, and what it won’t do.

With this research study, ORS Impact and Spark Policy Institute are seeking to understand the degree to which the collective impact approach contributed meaningfully to observed positive changes in people’s’ lives (or, in some cases, species or ecosystems).  In other words, when and under what conditions did collective impact make a difference where we’re seeing positive changes, or are there other explanations or more significant contributors to identified changes?  While we’ll learn a lot more than just that, at its heart, that’s what this study will do.  

Our primary approach to understand the core question around contribution and causal relationships will be to use process tracing.  Process tracing provides a rigorous and structured way to identify and explore competing explanations for why change happens and to determine the necessity and sufficiency of different kinds of evidence to support different explanations that we’ll find through our data collection efforts.

To implement the process tracing, we will dig deeply into data around successful changes—a population change or set of changes plausibly linked to the CI efforts—within six sites.  We’ll explore these changes and their contributing factors with data from existing documents, interviews with site informants, focus groups with engaged individuals, and a participatory process to review and engage in sense-making with stakeholders around the ways in which we understand change to have happened.  We’ll try and untangle the links between implementation of the collective impact approach and early outcomes, the links between early outcomes and systems changes, and the links between systems changes and ultimate impacts.

Figure:  Diagram of “Process” for Tracing

Note:  Future blogs will provide more information on the different rubrics we’ve developed and are using.

Using a process tracing approach also means that we’ll explicitly explore alternate hypotheses for why change happened—was there another more impactful initiative?  Was there a federal funding stream that supported important related work?  Was there state policy that paved the way that was unconnected to stakeholders’ work?  Would these changes have occurred whether collective impact was around or not?

Additionally, we’ll look at two sites where we would expect to see change but don’t, with the expectation that these sites can help us understand if the patterns we’re seeing at successful sites are absent or showing up differently, findings that would help give us more confidence that the patterns we’re seeing are meaningful.

Process tracing as our approach does mean that our unit of analysis—the sphere within which we will be exploring change and causal relationships—is going to be approximately eight sites.  While we hope to find sites where a cluster of impact outcomes result from a specific set of activities (or “process”), we are choosing to go deeply in a few sites with an approach that will provide rigor around how we develop and confirm our understanding of the relationships between activities and changes.  And because we are looking across diverse sites, working on varied issue areas (e.g., food systems, education, environmental issues, etc.) and at different scales (e.g., cities, multiple counties, entire states), identifying patterns across diverse contexts will increase our confidence around what collective impact conditions, principles and other contextual factors are most related to these successes.

With more data around if and when we find causal relationships, we will also go back to our data set of 22 sites that we are also engaging with early to see if we can, likewise, find similar patterns to those found through the process tracings.  For these sites, we’ll use data we will have collected on their fidelity to collective impact, efforts around equity, successes with different types of systems changes, and types of ultimate impacts.  Are we seeing similar patterns around the necessity of fidelity to certain conditions?  Are we seeing similar patterns in the relationship between certain types of systems changes and impacts?

Despite the strengths we believe this study has, it will not be the end-all-be-all, final say on the efficacy of collective impact.  All studies have limitations, and we want to be clear about those as well.  Given time and resources, we can’t conduct in-depth evaluations of the full range of efforts and activities any given collective impact site is undertaking.  Our unit of analysis isn’t a full site; it won’t take in the full complexity of the history of the initiative, or the full array of activities and efforts.  For example, it’s likely that a site that we engage with around a particular success has also experienced areas with no discernable progress.  We also are not comparing collective impact to other change models.  That doesn’t make the exploration of causality around successful changes less meaningful, but it does mean that we’ll understand contribution to specific changes well rather than understanding and judging the success of collective impact at a community-level or comparing collective impact to other models of driving systemic change.

We do believe that this study will fill a gap in the growing body of research, evaluation and evidence around collective impact by deeply understanding contribution in particular cases and by looking at a diverse and varied set of cases.  The social sector will benefit from continued interrogation of collective impact using many methods, units of analysis and approaches.  In the end, the more we learn, the better we can make meaningful progress on the gnarly issues that face vulnerable places and populations.

Posted on

Sparking Social Change

No More: Together we can end domestic violence and sexual assault

Rose Andom Center: one place, immeasurable hope

By Laura Trent, Esq. and Alison McCarthy, MSW

For many of Spark team members, their commitment to social change finds its way across their professional and personal life. For Laura Trent and Alison McCarthy, project managers at Spark, this commitment means volunteering for organizations that support survivors of interpersonal violence.  Laura is on the Board of The Rose Andom Center Young Professionals Council (YPC), supporting the development of The Rose Andom Center, which brings community organizations and government agencies together under one roof to provide for the needs of individuals and families affected by domestic violence. Alison is a hotline counselor for Denver-based sexual assault prevention and support center, The Blue Bench.

Blue Bench

April is both sexual assault awareness month and national volunteer month, so we wanted to take some time to bring attention to important issues and encourage you to find ways to “give back” or contribute meaningfully to your community by tapping into the issues you care most about.

The Issue

Interpersonal violence, especially violence against women, occurs at a staggering rate. In the US alone, conservative statistics estimate 1 in 5 women will be raped at some point in her life. Every minute, twenty people are victims of physical violence by an intimate partner. Though these issues have garnered national attention in the past few years (see domestic violence and the NFL, the White House’s “It’s On Us” campaign), there is much work left to be done to prevent this type of violence and to support the survivors who have been victimized.

Why We Volunteer

Alison: When I was figuring out what I wanted to do, career-wise, I thought a lot about the ways in which I could be a part of lasting, positive change for marginalized and oppressed populations. At the time, I was working as a legal advocate for domestic violence survivors and volunteering on a sexual assault hotline. As much as I loved that work and cared about my clients, I couldn’t shake the feeling that unless I worked to solve the systemic issues, there would always be another caller on the line or client in my office. So, I decided to pursue community social work to take on the large-scale policies, processes, cultural norms, and beliefs that hard people every day – that’s why I work at Spark, because that’s what I get to do every day. But it’s important to me not to forget that while I have the luxury of working at the 10,000 foot level on systemic change, there are real people who face tremendous hardship at the hands of those systems every day. It’s their voice and their stories I carry with me while I work, a reminder to keep going when the problem at hands feels too big and too complex to solve.

Laura: The concept of “giving-back” was never a concept for me growing up, in my home it was more of an expectation. However, my own relationship to philanthropy and how I perceive it was lacking in substance until I began to see the impact of singular actions between people. The interrelations among people and the social injustices I witnessed around me was, and still is, both weighty and often intimidating, but I find myself drawn more and more to this algorithm of society. This intrigue and passion led me to philanthropy, which literally means the love of humanity. My passion for philanthropy by happenstance led me to a unique opportunity with the Rose Andom Center, when I moved to Denver: a chance to help provide hope to those who simply need a new direction, which ultimately is hope, a change in direction, a new way. Similarly, in my work at Spark I get to see different directions implemented and adapted based on the paradigm encountered. The interweaving, inter-logging, and mixing of social responsibility across my professional and personal life invigorates my spirit and at the end of the day “the world will not be saved by the internet, but by the human spirit.” – Dr. Sherwin Nuland


No More: Together we can end domestic violence and sexual assault

Posted on

Oh DEAR – Why Experimentation is Key to Social Innovation

In preparing for a presentation at a collective impact forum in Oklahoma, I was trying to decide what I could share with a room full of social innovators that they wouldn’t already know. After seeing the line-up –including foundations, collective impact initiatives, community and political leaders – it was clear these were people with tremendous knowledge and experience. I suspect, though, we all have at least one thing in common: at some point along the way, we’ve all been involved in social change initiatives that didn’t change the world.

How many of us have participated in big comprehensive planning efforts, then went about implementing the plan and had little success, found more barriers then opportunities, gradually lost momentum and funding, and finally gave up?

What I realized is that I could share is something we all know, even if only intuitively. These comprehensive approaches aren’t how we cause meaningful change – not in this messy, complex world we live in today.

So what’s the alternative? It’s something we all do naturally and have done since we were little kids trying to understand the world around us. Experimentation. Testing. Taking small risks instead of big ones. To try to do good just a little bit better each time.

With this in mind, I decided to talk at today’s convening about a simple way of moving from “oh dear, that didn’t work” to “yes, we can do it (one little piece at a time)”. Let’s call it the DEAR framework for social innovation:

  • DEAR ModelDiscover: Explore where you are now, where you want to go, how you might get there. Use tools like theories of change and roadmaps and develop them quickly, in two or three meetings, not a year of deep thinking. Draw on needs assessments, existing data, community focus groups and more. Do it all in three months!
  • Experiment: Experiment with small changes you can cause along the way (sprints instead of marathons). Maybe try out a shift in how collaborative partners make decisions about when and how to deploy their services.  Or try out a small public private partnership to see if a market-based model can achieve social change. Know the outcome you’re trying to achieve with your experiment.
  • Assess: Take time to learn about what’s working. Are you moving the needle against that long-term aim? Use evaluation (but only if it’s quick!), rapid collection of data and intuitive learning to figure out what is working and what isn’t.
  • Replicate: Scale what’s working by trying it in similar settings, with different people, or just doing it more often. Keep discovering, experimenting and assessing around the areas where you haven’t found something that works.

These four steps came from reflecting on how some of the most exciting, successful social change strategies I’ve seen have actually functioned. They had big visions, but small actions. They had massive problems to solve, but took little steps along the way. They gave up on believing they could plan the road ahead and instead decide to walk it.

So, let’s try the DEAR model of social change as part of whatever collaborative, visionary work we’re already doing. Let’s discover, experiment, assess, replicate, and then share what we learn.

Oh dear, imagine the potential of what we can all do together if we just start small!