Posted on

How do you know if you’re getting the best quality in your evaluations?

How do you know if you’re getting the best quality in your evaluations?

RigorQuality in evaluation used to be defined as rigor (and sometimes still is), with rigor meaning the competence of the evaluator, the legitimacy of the process and, of course, applying the best research methods to the collection and analysis of data. These are important, but they don’t count as an all-encompassing definition of quality, particularly in complex, adaptive settings where evaluation partners with strategy.

If we cannot count of these measures to define quality, what are alternative ways of understanding if your evaluation is high quality? Hallie Preskill from FSG and I will be joining forces at the American Evaluation Association’s annual conference this Friday to explore this issue. We are proposing that the concept of “rigor” (and thus what you can look for in your evaluations) can – and should – be redefined as:

  • Balancing whether the evaluation is useful, inclusive of multiple perspectives, unbiased, accurate, and timely.
  • The quality of the learning process, including whether it engages the people who need the information when they need the information.
  • The quality of the thinking, including whether the evaluation engages in deep analysis, seeks alternative explanations, situates findings within the literature, and uses systems thinking.
  • The credibility and legitimacy of the findings, including whether people are confident in the ‘truth’ being presented.
  • Responsiveness to the cultural context, including the integration of stakeholders’ values and definitions of success, as well as who helps to interpret the findings.

Capture

Are you attending the annual conference? Come join us for an interactive discussion on how to reframe rigor and quality in your evaluations.

Posted on

Presenting Data Effectively

XKCD Convincing Comic

In my job as a Policy and Communication Manager for Spark, I spend a lot of time thinking about how we can translate the great research we do here into actionable products. When not presented effectively, data can be mostly noise – too much information to remember, interpret and use to make decisions. But, when data is presented effectively and at the right time, it can be a powerful tool for change. Here are some of the key things we use to guide our work here at Spark:

Be Clear on the Purpose: Is your primary goal to improve a strategy? To define a problem? To persuade? To educate? To call people to action? The same data can be presented in multiple different ways, which vary depending on why you are sharing the information. For example, if you’re looking to examine whether your strategy needs a refresh, you will need to focus on the data that is most actionable, whereas if you are looking to educate, there may be a need to provide additional context.

Know Your Audience: Does your audience have a depth of knowledge on the topic at hand or are they relatively new to the issue? Are you preparing a single document or presentation for an audience with varying levels of knowledge, interest, and/or education? Is their time limited? Do they need the facts quickly? Or will they have the time to digest the information more fully before taking action? Think about what is important to your audience and what they need to know the most. All of this will factor into determining how best to present information, e.g. an informal or formal presentation, a facilitated dialogue, or written, such as a brief, check-list, report, or white paper. These considerations will also influence the structure of the information and your tone.

Tell the Story: Stories make information more relatable, which makes your findings more actionable. Even if you don’t have a “traditional” story, it can be helpful to think of how you present your information using the elements of a story: characters (who is affected and who are the key players?), the setting (describe the context), plot (there should be a beginning, middle, and end), conflict (what is the central problem to be addressed and why is it important?), and resolution (what are some potential solutions, where do we go from here?). We tend to only remember one or two key things after a presentation or reading a document. If you present a series of findings, your reader will remember a couple of those findings, but will lose the rest. If you present a story, your reader will remember the overall picture of what the findings collectively mean.

Be Visual: Most of us are, by our nature, visual creatures. Often, showing is more effective than telling at conveying information, which means you’re increasing the likelihood your findings are used! Visuals can include charts, graphs, tables, maps, infographics, even pictures. Taking the time to think through which type of visual is the most appropriate for your audience, as well as what results are most worth highlighting, can help ensure your visuals have the most impact.

 

XKCD ConvincingActively Engage: Sometimes you don’t have direct access to your audience in-person or by phone. Instead, you’re sending documents, posting them online, etc. When you do have access to the audience, however, you can take the use of data to the next level. When we are presenting information, whether virtually or in the room, we don’t do half hour long PowerPoint presentations. We talk for 10 minutes, discuss and debrief for more than 10 minutes and repeat. We put the data up on the walls and let people walk around the room and talk with each other. We put visuals and stories on the tables and have small groups explore the meaning. We find ways to make sure that more time is spent digesting and thinking about how to act on the findings than spent absorbing the new information. When people integrate new information into how they are thinking about a problem or a solution, they are more likely to use it than when they simply learn the new information.

There are many great online resources out there for presenting data effectively, including our recently-released toolkit in our Tools for Social Innovators series, Data as a Tool for Change. We’d love to hear how you incorporate data into your work, and tips and tricks you have for presenting information effectively. Share in the comments or click here to contribute to the toolkit!

Posted on

Introducing Spark Policy

At Spark Policy Institute, we are dedicated to helping communities and policymakers solve complex problems.  The Spark blog will focus on concrete strategies and actions you can take as you seek to make a meaningful difference on issues that are challenging, complicated, and critically important to you and your community.

 

What are complex problems?

Called “intractable problems” by some, complex problems have a mix of stakeholders at all levels of government, each of whom have different funding sources, mandates, and expectations; these problems also have private stakeholders, consumers, and communities that cannot be left out.

What are some examples?

  • As regards the obesity epidemic, finding the policy and individual behavior change solutions that can reduce obesity and its associated illnesses;
  • With healthcare access and reform, moving beyond playing politics and into making a difference;
  • With water policy in the arid west, finding solutions that balance environmental, agricultural, and population needs;
  • Healthy food access and its intersection with transportation infrastructure, water, land-use planning, nutrition education, and schools; and
  • Behavioral health and its intersection with all aspects of our lives, from workplace productivity to juvenile justice involvement to physical health outcomes.

Spark’s work is dedicated to the challenge of addressing complex problems such as these, bringing together a combination of research, engagement of all stakeholders, and information dissemination to help find solutions.

 

What do we know about these complex problems?

We recognize that regardless of which policy arena a problem emerges from, common issues are often present:

  • Policy solutions identified in one arena are likely to cause unintended consequences in others;
  • Money talks – part of identifying any policy solution is understanding how public and private funding operates, what the limitations are, and where to find opportunities to leverage;
  • Sustainable solutions and change in the status-quo are only successful when a wide range of stakeholders are involved in identifying both the problem and the solution; and
  • Finding solutions is only the beginning – implementing change is a long, slow process that requires commitment, resources, regular evaluation and feedback, and engagement of all the stakeholders.

 

How do we solve these complex problems?

As Spark has grown, we have built skills and expertise to tackle complex problems in a wide variety of arenas: human services, health, behavioral health, natural resources, agriculture, housing, juvenile justice, criminal justice, education, early childhood, and diversity / disparities.

The Spark team we have assembled over the years now includes a mixture of:

  • Researchers who are adept at working in messy, complex settings and bring a wide variety of methodologies to their work including fiscal and legal research, evaluation, network analysis, q-methodology, focus groups, and many other quantitative and qualitative approaches;
  • Facilitators who understand how to inform dialogue with external information and input, and can create a safe environment where all stakeholders, including community members, consumers, and even youth, can participate fully in complex policy dialogues;
  • Project managers whose approach reflects the needs of their clients, and who can remain flexible as the policy environment changes; and
  • Product developers, who specialize in ensuring reports, white papers, presentations, and other materials are rich in information and attractive in presentation, but more importantly, are committed to making sure no product becomes yet another report that sits on a shelf.

 

What does a “solution” look like?

Every problem has a different solution, and we know that the solutions that are first tried often fail to address fully all the complexity of the problem.  What does a solution look like?  There is no single answer – every system is different.  Maybe the solution includes changes in how funding is utilized by government agencies.  Maybe it includes changes to policies related to access to care.  The solution might be about how non-profits mobilize and educate their communities.  It may also include new voices having a say in the decision-making process.  Sometimes a solution is about changing how existing policies are implemented and sometimes it requires an overhaul of laws and regulations.

 

Join Us

Join the Spark Team in our dedication to solving complex problems.  What are the issues facing your community?  How can you tackle them?  Each week, the Spark blog will release new tips, tools, research, and information to help you find those solutions.

Do you have any questions you want answered?  Please let us know the topics you want to learn more about!