Measuring and learning ... for improvement

The challenges of measurement. When community groups design programs, evaluation is usually the last item on the agenda. The same can be true for community giving as well.

There are many reasons for this. We all know that evaluation has negative connotations. It conjures up memories of the day we received our SAT results. We associate it with grades and scores. We know what these experiences can be like:

• They assume wrongdoing
• They are only quantitative
• They are mainly about control
• They produce anxiety about how we measure up

Evaluation can conjure fears in nonprofit staff, and perhaps you too. The fears may include:

• I don't speak the evaluation language
• I am not a numbers person

Or that the evaluation will:

• Drive the focus of the program
• Make recipients anxious
• Make me or the foundation look bad
• Distract recipients from the real work
• Inhibit our entrepreneurial and innovative nature
• Mean reading huge reports

These fears can make us reluctant to pursue evaluation. Recently, however, new trends in thinking about and using evaluation have evoked more positive responses. When handled well, measuring results and evaluating performance can have powerful effects.

When to evaluate. You may be wrestling with the question of when it is appropriate to invest time and resources in evaluation. As foundations of all sizes and types have developed and tested approaches to evaluation, a few lessons have emerged:

• Evaluations should be anchored in clear programmatic goal statements. Your ability to learn from an evaluation depends on the degree to which goals and results are clearly articulated, measurable and feasible.
• Programs operate in complex social environments. Your grant may be part of a larger set of factors and circumstances that contribute to individual or community change. It can be very difficult to “prove” that results are directly attributed to your giving. You are better off using measurement and evaluation to help you and your recipients understand how the results were achieved or why they were not achieved, and moving beyond both credit and blame.
• Programs should achieve a certain threshold of maturity and stability before useful conclusions can be drawn. Identifying intermediate steps or milestones is a useful method of measuring progress and determining where that threshold might be. 
• Evaluation demands a significant amount of preparatory work. Identifying and hiring an external evaluator; specifying clear, measurable goals; and developing an evaluation plan can be time-consuming but will be critical to creating evaluations that will be insightful and practical.
• Recipients are more likely to benefit from evaluation when they have been actively involved in the evaluation process, its design and goals. Evaluators should involve them and reinforce the intended use of evaluation at every step along the way.

Evaluations are only useful when you, your fellow trustees and staff are receptive to outside scrutiny and feedback.

An evolving definition and purpose. As a discipline, program evaluation grew out of the massive government programs of the 1960s and 1970s. Policymakers and bureaucrats turned to evaluation as a way to judge whether a program was successful and worth funding.

Evaluation, however, did not provide the clear answers that decision makers desired. Evaluators complained that findings were ignored. In practice, evaluation was used by funders more as a control mechanism than as a learning tool. As a result, service providers became more focused on compliance than results and were rewarded for doing paperwork rather than making a difference in communities.

In response, the concept of evaluation is shifting within the foundation field. Rather than simply using evaluation and measurement to determine whether grantees did what they said they would do, foundations now integrate measurement and evaluation into core planning processes. [See So What? from Designing a giving program]

Before embarking on any evaluation, you should address two fundamental questions:

• What do you want to know?
• Who will use that information, and how?

This sequence is important, because the methods you use to measure and evaluate should be chosen with purpose and audience in mind. Regardless of the size and mission of your giving, chances are you are measuring results to serve one or more of three general purposes:

• Accountability: to measure the results of programs and account for use of resources
• Knowledge generation: to create new understanding about what works and what does not with a particular program
• Future program planning or improvement: to support future planning, implementation, and overall effectiveness

In terms of audience, you may want to consider one or more of the following audiences:

• Foundation: board, staff (if you have any), family members
• Grantee: board, staff of the recipient organizations, community residents
• Field: practitioners in the same field, fellow donors, consultants, academics, policymakers

While each purpose and audience is legitimate, each also represents different ways to think about and organize an evaluation. They differ in regard to the questions that would be posed and the methods and evaluator competencies required to address those questions.

Priority Audience	Intended Use	Key Questions
You (the donor or your staff)	Program improvements Learning Accountability	·        Was the program implemented as intended? ·        How can implementation be improved? ·        What external issues affected the program? ·        How might they be addressed? ·        Is the strategy still relevant and viable? ·        How can the program be improved? ·        Did the program reach the target population? ·        Did the program achieve its intended results?
Fellow directors, trustees or family members	Accountability Learning	·        Did the program achieve its intended results? ·        Did the program contribute to the mission of our giving? ·        Is the strategy still relevant and viable? ·        Is the program worth the cost?
Recipients	Program improvement	·        Was the program implemented as intended? ·        How can implementation be improved? ·        How can program efficiency be improved? ·        Is this strategy viable? ·        How can successes be communicated to others? ·        How can additional resources be attracted?
Policy community	Accountability Learning	·        Whom did the program serving? ·        Did the program reaching its target population? ·        What did participants think about the program? ·        Did the program achieve its intended results? ·        Was the program cost-effective? ·        Is this strategy viable? ·        In what context was the program most effective? ·        How did the program fit with other community efforts?
Local community members	Accountability	·        Was the program implemented as intended? ·        What did the program accomplish? ·        Was the program suited to our community's needs? ·        How does the program compare to others in the community?

You may find that you simply do not need a professional evaluator because the scale and complexity of your giving have not reached a certain level. In that case, you may decide that you, a staff person and your recipients are in the best positions to answer the evaluation questions most important to you.

If so, use the table above to help you determine which questions will need to be answered to meet the needs of your intended audience(s). When compiling answers, remember to consider your own biases and those of the folks you question. Are you able to maintain the proper objectivity to honestly and effectively measure the results of your giving? If so, carry on.

On the other hand, you may find yourself injecting your own perspective into the questions and answers. You might find recipients less willing to speak with you than with an impartial outsider. You may find your fellow trustees more receptive to an outside evaluator's judgments or that your giving is just too complex to evaluate on your own.

Measuring Results was adapted from When and How to Use External Evaluators (2002), by Tracey A. Rutnik , Director, Funders Evaluation Initiative, Association of Baltimore Area Grantmakers, and Martha Campbell, Vice President for Programs, The James Irvine Foundation. We gratefully acknowledge the authors' willingness to share their work with the Community Giving Resource.

The following tips can help you find and hire an outside evaluator or, alternatively, can help you judge whether you will be able to play this role effectively. 

The second attachment Evaluation Terminology is a useful, straightforward guide to the sometimes confusing vocabulary of evaluation.

Download Attachments

Hiring and Managing Evaluators (7022K) [description]    [download] Evaluation Terminology (7014K) [description]    [download]

Related Reading

Resources

Making a Difference: The Philanthropic Initiative (TPI) offers simple, pragmatic approaches to evaluation.

Evaluation: The Good News for Funders