Posted on

Embracing Values in Evaluation Practice

Research has traditionally defined rigor as obtaining an unbiased estimate of impact, suggesting the need for experimental or quasi-experimental methods and objective, quantitative measures in order to obtain trustworthy results.

I’ve spent the past few months as a member of Colorado’s Equitable Evaluation Collaboratory, which aims to examine the role evaluation plays in supporting or inhibiting progress toward equity and identifying opportunities to integrate equitable evaluation principles into practice. In particular, I’ve reflected on how the research tradition has impacted evaluation’s working orthodoxies including the notion that “credible evidence comes from quantitative data and experimental research” and “evaluators are objective.”

On the surface, these statements don’t appear particularly problematic, but dig a little deeper and we begin to see how value judgments are an integral part of how we practice evaluation. The types of projects we take on, the questions we ask, the frameworks we use, the types of data we collect, and the ways we interpret results – are all deeply rooted in what we value. As an evaluator focused on use, I aim to make these practice decisions in partnership with my clients; however, suggesting that I, or any evaluator, does not play an active role in making these decisions discounts our inherent position of power.

Now that I’ve tuned into the orthodoxies, I see them everywhere, often dominating the conversation. In a meeting last week, a decision-maker was describing the path forward for making a controversial policy decision. He wanted to remove subjectivity and values from the conversation by developing guidelines rooted in “evidence-based practice” and turned to me to present the “facts.”

As a proponent of data-driven decision making, I value the role of evidence; however, there is a lot to unpack behind what we have declared – through traditional notions of rigor – “works” to improve health and social outcomes. Looking retrospectively at the evidence, and thinking prospectively about generating new knowledge, it’s time to ask ourselves some hard questions, including:

  • What interventions do we choose to study? Who developed them? Why did they develop them?
  • What have we (as a society) chosen not to investigate?
  • What population have we “tested” our interventions on? Have we looked for potentially differential impacts?
  • What outcomes do we examine? Who identified these impacts to be important?
  • Who reported the outcomes? Whose perspective do we value?
  • What time-period do we examine? Is that time-period meaningful to the target population?
  • Do we look for potentially unintended consequences?

As we begin to unpack the notion of “what works” we begin to see the decision-points, the values and the inherent power and privilege in what it means to be an evaluator. It is time that we owned the notion that what we choose to study and how we choose to measure success are not objective, rather, they are inherently subjective. And importantly, our choices communicate values.

So how do we begin to embrace our role? As a step forward, I have started including a discussion of values, both mine and my clients, at the beginning of a project and clarifying how those values will influence the evaluation scope and process. Explicitly naming the importance of equity during the evaluative process has helped keep the goals of social change and social justice front and center.  Naming values helps stakeholders acknowledge their power and provides a lens through which to make decisions.

Equitable evaluation is an expedition into the unknown, requiring a transformation in how we conceptualize our role as evaluator. Having taken my initial steps into the Upside Down, I look forward to the many unknowns.

In what way do you see values showing up in your evaluative work?

 

Posted on

Diversity, Equity, and Inclusion at Spark

“I raise my voice not so that I can shout, but so that those without a voice can be heard.”

― Malala Yousafzai


Dear partners and friends,

Spark Policy Institute was founded with a core belief that diverse perspectives are key to achieving meaningful change. Each of our conference rooms are named after leaders who made a difference because they spoke up and spoke out: Malala Yousafzai, Rodolpho “Corky” Gonzales, Hattie McDaniel, Maya Angelou, Dolores Huerta, and Sojourner Truth.

Our role as a bridge between nonprofits, communities, and funders brings with it a great responsibility to ensure all voices are heard in strategy design, implementation, research, evaluation, and interpretation. With the mark of the new year, we are committed to assessing how we are doing and where we need to improve. One of the ways we are doing that, is by renewing our focus on the issues of diversity, equity, and inclusion (DEI).

Our team is taking on the hard work to make this vision reality – we have assembled not just a committee, but a working group, whose key charges are to develop goals for improving our work and our internal culture to more truly and demonstrably respect diversity, value equity, and foster inclusion.

We are committed to sharing our journey with specifics about the steps we are taking to reach our goals. You can read an introduction to our reinforced effort in our latest blog post Journey to Diversity, Equity, and Inclusion, which includes a link to our DEI Commitment, and our 2018 DEI Plan. We will continue to update our website and newsletter with all of our efforts.

I am sincerely humbled by the skill and perseverance of my colleagues who continually seek to improve our work and challenge our assumptions. Together in this process we are capable of remarkable change.

As always, we at Spark welcome your questions, suggestions, or reactions. Please do not hesitate to reach out to me, or any of our staff as we move forward with this critical work.

Sincerely,
Kyle Brost
CEO
Spark Policy Institute

Posted on

Journey to Diversity, Equity, and Inclusion

Alison McCarthy

Capitalizing on the new year’s encouragement of new habits and improvement, we are excited to announce a renewed focus to build and improve on Spark’s Diversity, Equity, and Inclusion (DEI) efforts. This undertaking reflects our organizational and individual staff values. Much like New Year’s Resolutions that last, we know this type of work requires ongoing commitment through successes and challenges, a humble and realistic approach, and support from those around us.

Spark has had an inclusiveness statement since 2014 and has used participatory research and facilitation approaches since its inception. The general sentiments underlying DEI have been a part of this organization since the beginning. We believe it’s time to build on these sentiments to operationalize them – to “walk our talk,” so to speak. The people we work with and for deserve it. Additionally, many of us are drawn to the work we get to do at Spark because of our desire to make a positive difference in the lives of others. To do so effectively, we know we need to work to better understand all the layers that make up people’s rich cultures and world views. It is the first step towards building the authentic relationships we seek to cultivate.

Our DEI plan, which you can see here, includes both internal goals for staff awareness and skill building and external goals around the way we work with stakeholders, partners, and clients. We aim to infuse DEI into our language, behaviors, tools, approaches, products, processes, and policies. Part of the reason for this range of focus areas is the recognition that both an individual and systemic approach are needed for real change.

This affects all of us at Spark, and our DEI team (Daniela, Alison, Kristin, and Adell) will be guiding the implementation of this plan. As a team, we largely reflect Spark in this process: passionate about our values and dedicated to this work, though by no means experts.

We commit to keep you updated on our progress. This will include both the successes we have and the challenges we bump up against. We’ll share lessons learned and highlight partners who are taking on this work.

We sincerely hope we hear from you, too. Have a resource to share? Got some advice or feedback for us? Are you struggling with something in this area where we can either help or brainstorm together? Let us know! We don’t have the perfect formula here, and we don’t expect everything to go smoothly. But we know something is better than nothing, so we’re doing our best and taking some important steps. We hope you’ll join us on this journey!

Posted on

Collective Impact Study Sites

Site/Initiative Name Location Issue Area Contribution Site Equity Site
Alignment Nashville Nashville, TN Education – Multi-Issue X
ARISE Anchorage, AK Education for Indigenous Students X
Aspen Community Foundation Cradle to Career Aspen, Basalt CO Cradle to Career – Education
Coalition for New Britain’s Youth New Britain, CT Cradle to Career – Education, Early Learning
Colorado Consortium for Prescription Drug Abuse Colorado Substance Abuse X
Communities that Care: Franklin County and the North Quabbin Franklin County, MA Education – Reducing Teen Social Risk Factors X
Connecticut Juvenile Justice Alliance Bridgeport, CT Juvenile Justice X
Elizabeth River Portsmouth VA Environmental X
Green Umbrella Greater Cincinnati, OH Area Food Systems; Environmental Sustainability
Home for Good Los Angeles, CA Homelessness X
K-Connect Kent, MI Cradle to Career – Education
Living SJ Saint John, New Brunswick, Canada Poverty Reduction
Metro Area Continuum of Care for the Homeless Omaha, NE Homelessness
Milwaukee Teen Pregnancy Initiative Milwaukee, WI Teen Pregnancy X
Mission: Graduate Central New Mexico Cradle to Career – Education
Open Doors of Fairfield County Fairfield County, CT Homelessness
Ottawa Child and Youth Initiative: Growing Up Great Ottawa, Canada Cradle to Career – Education, Early Learning
Project U Turn Philadelphia, PA Opportunity Youth
Promesa Boyle Heights Los Angeles, CA Cradle to Career – Education X
RGV Focus Rio Grande Valley, TX Cradle to Career – Education X
San Diego Childhood Obesity Initiative: Community Health Improvement Partners San Diego, CA Health – Childhood Obesity X
Saskatoon Poverty Reduction Partnership Saskatoon, Saskatchewan, Canada Poverty Reduction
Shaping our Appalachian Region (SOAR) Southeast Kentucky Economic Development
South Platte Urban Waters Partnership Golden, CO Environmental
Vermont Farm to Plate Vermont Food Systems

 

Posted on

Using frameworks to communicate systems level insights

Collaborative Framework to improve school attendance

Laura Trent, Esq., Senior Consultant at Spark Policy InstituteThe Juvenile Justice and Delinquency Prevention (JJDP) Council has a tall order – prevent youth from entering the justice system or from penetrating deeper into the justice system. The goal cannot be met through program implementation alone. The systems serving the youth must change how they work together (or begin working together) in order to meet the needs of “at-risk” youth.

In 2013, the JJDP Council recognized the need for an alternative approach to addressing “status offenses” (only a crime because of a youth’s age) like truancy (missing too much school). They funded four truancy demonstration pilots in Colorado to work across systems in order to prevent and intervene in truant behavior.

Our new report, Evaluation of Truancy Prevention and Early Intervention takes a retrospective look at the four pilots and includes reflections from stakeholders in schools, courts and the justice system. This led to our Collaborative Framework to Improve Educational Attainment (below), which identifies critical areas and components Colorado’s broader juvenile field must address to work together to improve school engagement.

Collaborative Framework to improve school attendance
This framework outlines the importance of partnerships, prevention, and intervention approaches, and sustainability in supporting Colorado youth and families in the education and justice systems.

At Spark, we understand the importance of connecting theoretical frameworks with on-the-ground  perspectives to ensure our work is actionable.

We facilitated a dialogue with the JJDP Council on the evaluative findings and asked whether it resonated, what was missing, and how it could be built upon. We were able to map the framework to the reality of the stakeholders needs because of our participatory approach (see Spark’s toolkit on Tools for Engaging Nontraditional Voices). Their input made clear that there is room for the framework to not only serve as a guide to improving school engagement, but more broadly to meet the needs of at-risk youth through a collaborative approach.

For more about the work of the truancy demonstration pilots and truancy in Colorado visit the Colorado Division of Criminal Justice website.