What is Impact Evaluation? (2024)

An impact evaluation relies on rigorous methods to determine the changes in outcomes which can be attributed to a specific intervention based on cause-and-effect analysis. Impact evaluations need to account for the counterfactual – what would have occurred without the intervention through the use of an experimental or quasi-experimental design using comparison and treatment groups.

Purpose of Impact Evaluation

Impact evaluations often serve an accountability purpose to determine if and how well a program worked. Impact Evaluations can also help answer program design questions to determine which, among several alternatives, is the most effective approach.

When to do an Impact Evaluation

The World Bank has developed the following guidelines for determining when an impact evaluation may be useful¹:

• If it is an innovative intervention scheme, such as a pilot program.
• If the intervention is to be scaled up or replicated in a different setting.
• If the intervention is strategically relevant and will require a great deal of resources.
• If the intervention is untested.
• If the intervention results will influence key policy decisions.

Design of Impact Evaluations

There are many design options for impact evaluations depending on a variety of factors (see USAID Technical Note and OECD Principles for Impact Evaluation for helpful decision trees for design). However, all designs generally fall into either an experimental or quasi-experimental design through the use of a counterfactual, an alternate control group generated by random selection in the case of experimental designs, or a comparison group which is not randomized in the case of a quasi-experimental design.

Common Experimental Designs include the following:

• Randomized Controlled Trial (RCT)
• Randomized Offering of Intervention
• Randomized Promotion of Intervention
• Multiple Treatment Design

Common Quasi-Experimental designs include the following:

• Difference-in-differences
• Matched Comparisons
• Regression Discontinuity
• Interrupted Time Series

Impact evaluations are not appropriate for every intervention scheme, as such evaluations require certain data sets and financial resources that may not be available. Evaluators usually work in conjunction with project planners and donors when proposing to conduct impact evaluation.

See the following list of resources for a more detailed and in-depth discussion of impact evaluation and available designs.

List of Resources

InterAction, Impact Evaluation Guidance Notes

OECD Principles of Impact Evaluation

USAID, (2013) Technical Note

World Bank (2011) Impact Evaluation in Practice

References:

¹ World Bank, Impact Evaluation in Practice http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1295455628620/Impact_Evaluation_in_Practice.pdf

About the Author

Kirsten Bording Collins is an experienced evaluation specialist providing consulting services in program evaluation, planning and project management. She has over ten years of combined experience in the nonprofit, NGO and public sectors working both in the U.S. and internationally. Kirsten's areas of expertise include: program evaluation, planning, project management, evaluation training and capacity-building, mixed-methods, qualitative analysis, and survey design. Kirsten holds a MA in International Administration from the Korbel School of International Studies, University of Denver. Kirsten grew up in Copenhagen, Denmark and currently lives in Washington, DC.

Connect with Kirsten on LinkedIn.

________
To learn more about American University’s online Graduate Certificate in Project Monitoring and Evaluation, request more information or call us toll free at 855-725-7614.