Examples of quasi-experimental designs include. Whenever a prospective impact evaluation involving treatment and comparison groups is being considered, it is wise to undertake apower analysisto ensure that the number of units (people, locations) available for assignment to these groups is adequate to detect important differences between them. The technical storage or access that is used exclusively for anonymous statistical purposes. It determines whether the intervention can be considered a success, an improvement or the best option. These are just a handful of commonly used impact evaluation methodologies in international development, to explore more methodologies, check out the, Choosing Appropriate Designs and Methods for Impact Evaluation.. COVID-19 Policy Impact Evaluation: A guide to common design issues Noah A Haber, Emma Clarke-Deelder, Joshua A Salomon, Avi Feller, Elizabeth A Stuart Policy responses to COVID-19, particularly those related to non-pharmaceutical interventions, are unprecedented in scale and scope. Additionally, your workers will feel heard and valued if you consult with them and, more specifically, if you act on a proposal they provide. UNDP OUtcOme-LeveL evaLUatiON: a cOmPaNiON GUiDe iii . Such impact evaluations can help you make decisions about program elements in a short timeframe. It involves consulting with key informants to identify either a hypothetical counterfactual, meaning what they think would have happened in the absence of an intervention or a logical counterfactual, meaning what would logically have happened in its absence. These observed changes can be positive and negative, intended and unintended, direct and indirect. Impact evaluation - Impact Measurement Guide Which version of my program is more effective? Picking up subtle changes (such as a small boost in test scores) requires a bigger sample than detecting large changes (such as mortality rates falling by half). There is usually a cut-off point to determine who is eligible to participate. PDF INTRODUCTION TO IMPACT EVALUATION - InterAction However, if you are seeking an accountability system, monitoring is better-suited as it is continuous, whereas a process evaluation is a one-time exercise. Ses consquences, en . Should you choose to collect data internally, you should attempt to minimize response bias by hiring external enumerators, especially if the evaluation is asking sensitive personal questions. note is a guide to implementing an impact evaluation for youth employment interventions, covering all aspects of the evaluation process from preparation to the dissemination of results. Think about the evidence you find as well-documented ideas, but not the last word. it has a large number of citations or is cited by many other studies in your review), it is relevant to specific aspects of this evaluation (such as measuring similar outcomes, being conducted in a similar context, or evaluating a similar intervention), etc. There are still 9 evidence reviews! Impact and evidence based planning, monitoring and evaluation is an intrinsic element of the CSIRO's Planning and Performance Framework. Guide. Under most circumstances, organizations opt to work with an external evaluator for an impact evaluation. Energy Efficiency Program Impact Evaluation Guide ", Stern, E. (2015). Impact evaluation will generally answer three types of questions: descriptive, causal or evaluative. Plan includes key elements, questions, methodologies & more. One method of achieving this is using a specific rubric that establishes various performance levels (or standards) for each evaluation criterion. Locate available resources and figure out how to deploy them. Both process evaluations and monitoring are implemented for learning is our program being implemented as planned? The effect evaluation approaches need a lot of planning, ample time for baseline data collecting, and, when necessary, the development of a randomized control trial or comparison group or the use of other strategies to look at causal attribution. However, some interventions cannot be thoroughly planned, such as programs in environments where implementation must react to new possibilities and obstacles, like supporting the enactment of legislation in a contentious political context. The intervention could be a small project, a large program, a collection of activities, or a policy. Even though they will receive a free replacement, they will still have to wait longer for the item. Impact Evaluation: Strategies, Examples & Differences Every development program rests on a theory of change its a crucial first step that helps you remain focused on impact and plan your program better. Outcomes as well as data of ranking criteria, e.g. Skip to content Features Data Collection Data Management Results Framework & Indicators Dashboards Does the program work in new contexts? The resulting treatment and non-treatment (or control) groups are deemed to be functionally equivalent, as all of the other characteristics of population members have been distributed across both groups. Required fields are marked *, A monthly round-up of news and useful links on the digitisationof the sustainable development sector, from the team at TolaData, This article is the second part of our 2-part series on impact evaluation. You should conduct a practical analysis and determine how to maintain the goods safely in transit if you sell a particular product, but half of them are destroyed in transit. Every impact assessment should utilize a theory of change in some capacity. PDF Outcome-Level Evaluation Guide - UNDP This toolkit from the World Bank provides a number of modules which guide the user on the design and implementation of impact evaluations. Start with the3ie Development Evidence Portal, which has compiled over 3,700 evaluations and over 700 systematic evidence reviews. This article is partly based on the Methodological brief, Outline of Principles of Impact Evaluation, Technical Note on Impact Evaluation USAID. 9. refers to the rationale for conducting an impact evaluation. Stakeholders will participate in the impact evaluation depending on these and other factors. Consider what decisions need to be made and the timing for the decisions. Which people do the results need to convince? Any developmental programs impact must be measured, which is challenging. You will likely still need to use Method 2 for finding more recent research as well as evidence from non-academic sources, Developing an evaluation design that is feasible given the context of your program. However, keep in mind it can take 1-2 years to write and publish a review and since the review is citing only published literature, the cited articles would be a year or two old as well. The guide signposts more specialistsources and references, but is mainly interested inequipping practical managers in the developmentsector with enough knowledge to allow them to havemeaningful conversations with technical experts. If a study mentioned in the review seem particularly relevant, then you will probably want to read it. Evaluation Criteria - OECD The summary will likely be useful but too vague dig into the review and look for details about which programs were tried and where, and how well they worked. BetterEvaluation is part of the Global Evaluation Initiative, a global network of organizations and experts supporting country governments to strengthen monitoring, evaluation, and the use of evidence in their countries. You can potentially shorten the evaluation timeframe by measuring intermediate outcomes, in cases where theres a well-established link between the intermediate and final outcome. Objective of evaluation; A model of change to evaluate; Different types of evaluation; Evaluation and cost benefit analysis; When to do an impact evaluation; Putting it all together: developing an evaluation strategy; Learning from impact evaluations Impact evaluations should be focused on key evaluation questions that reflect the intended use of the evaluation. The Impact Measurement Guide is still in beta release, we welcome your feedback on which features you like as well as what we should change or improve in future versions. What is an impact evaluation? | AidGrade Getting intermediate results may help create the desired ultimate impression. To answer these questions one needs to identify criteria against which to judge the evaluation results and decide how well the intervention performed overall or how successful or unsuccessful an intervention was. Like other grant-making organizations, the Open Society Foundations achieve impact largely through the work of its grantees. In practice, it is quite complex for an organisation to fully claim attribution to a change, this is because changes within the community are likely to be the result of a mix of different factors besides just the effects of the intervention, such as changes in economic and social environments, national policy etc. As mentioned above, many impact evaluation uses mixed methods, which is a combination of qualitative and quantitative methods. A monitoring system collects real-time data about your programs performance. A lock () or https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely. For all evaluations, carefully selected and correctly used data gathering and analysis procedures are crucial. What function would stakeholder involvement in this impact evaluation serve? While data on community needs and program performance is important and useful, there are situations when an impact evaluation is absolutely the right tool for the job. The GEI focuses support on efforts that are country-owned and aligned with local needs, goals and perspectives. Measuring direct causes and effects can be quite difficult, therefore, the choice of methods and designs for impact evaluation of interventions is not straightforward, and comes with a unique set of challenges. service providers or locations)that need more attention. PDF | On Jan 1, 2014, Gustavo Angeles and others published A Guide to Longitudinal Program Impact Evaluation | Find, read and cite all the research you need on ResearchGate An impact evaluation must determine the causal attributionthe reason for the observed changes. Designing an impact evaluation work plan: a step-by-step guide For more information, please seeRunning Randomized Evaluations, Module 4.2:Choosing the Level of Randomization. 1: What is impact evaluation and when should you do it? The more certain we are that groups are equivalent at the start, the more confident we can be in claiming that any post-intervention difference is due to the project being evaluated. Data collection options include, but are not limited to interviews, questionnaires, structured or unstructured and participatory or non-participatory observations recorded through notes, photos or video; biophysical measurements or geographical information and existing documents and data, including existing data sets, official statistics, project records, social media data and more. This will help you design an impact evaluation that answers your questions while also keeping your constraints in mind: Which decisions will your organization make on the basis of an impact evaluation? In other words, they are not limited to collecting and analyzing quantitative or qualitative data, nor are they confined to particular evaluation methodologies. Search using keywords related to your problem, 4. Evaluation workplan: criteria for selection of evaluation topics 109 Annex 10 Checklist for evaluation terms of reference 113 Annex 11 Methodological approaches to impact evaluation 121 Annex 12 Core competencies for evaluators 123 Annex 13 Evaluation workplan template 125 Annex 14 . One should have a clear understanding of the objectives and assumptions of the intervention, what baseline data exist and are available for use and what new data needs to be collected, how frequently, in what form, and what data do the beneficiaries need to deliver etc. It is ethical to provide a free replacement when anything is harmed in transit. Impact evaluation | Better Evaluation How will you use the results? This report provides a guide to evaluating the impact of a project or programme. Our intention behind the 2-part series was to explain impact evaluations and its key components in a simple manner so that you can plan and implement your own impact evaluation more accurately and effectively. when experimental and quasi-experimental designs are not possible, we can conduct non-experimental designs for impact evaluation. ToCs should be reviewed and revised on a regular basis and kept up to date at all stages of the project lifecycle be this at project design, implementation, delivery, or close. Say you conduct an impact evaluation of a vocational training program. In the first article, , identify the real difference your interventions are making on the ground, The purpose, objectives and scope of the evaluation, Key evaluation questions the evaluation aims to answer. Chapter 2 of the PME Handbook and, in particular, Section 2.4 . by Erika Hernandez | Aug 27, 2022 | Monitoring and Evaluation, Social Impact, Surveys, Technology. Six months after finishing the program, a majority of participants were employed. To guarantee that the requirements are fully addressed, each evaluation question should be directly linked to the evaluative criteria. In todays immediate gratification environment, most people are not inclined to do this. We hope you found this article helpful. It is important for those who commission, manageand use Impact Evaluations to have access toframeworks and guidance. This results in an impact evaluation that is classified as having aquasi-experimental design. COVID-19 Policy Impact Evaluation: A guide to common design issues By its very nature, evaluation provides solutions to issues concerning worth and quality. It helps assess whether the objectives of a specific intervention were achieved and also measures unintended consequences. Now read the titles of each review and start going through the ones that seem applicable to you. Try searching more about the program and you might find something. For instance, it is choosing to scale up a program that is ineffective or effective only in a few particular circumstances or ending a program that could improve if addressing limiting constraints. The segment shaded in orange shows the extent to which the group that received textbooks . The central scope is usually specified in the work plan or the terms of reference (ToR) and the extended scope in the inception report. You may also learn that an impact evaluation is not suited to answering your question for example, it is difficult to evaluate systems-wide change, or a program that is very small. You probably dont need to read all of it the abstract, introduction, a description of the program, the results, and the conclusion are probably enough, whereas sections on technical methods can be skipped. This can be individuals, schools, villages, etc. This blog post will discuss the different types of impact evaluations and tips on planning an impact evaluation study. How did the intervention work in conjunction with other interventions to achieve outcomes? the areas of Evaluation Research, Social Accounting, Eco logical and Social Impact Assessment, NPO research, Social Entrepreneursh ip as well as Profit-oriented Entrepreneurship, especially regarding A ToC may also be represented in the form of a logic model or a results framework. Impact evaluations can be divided into two categories: prospective and retrospective. In the first article, Impact evaluation: overview, benefits, types and planning tips,we introduced impact evaluation and some helpful steps for planning and incorporating it into your M&E plan. For example, if you find that your vocational training program does not cause improved job outcomes, you would want to consider redesigning and retesting the program, or possibly terminating the program altogether. Impact Evaluation Designs | Project Starter | Program Cycle | U.S It is an in-depth, one-time exercise that can help identify gaps in your program. A ToC also helps to identify gaps in logic or evidence that the evaluation should focus on, and provides the structure for a narrative about the value and impact of an intervention. Showing that your program works would help you demonstrate to your donors that the funding they provided led to positive impact. Both experimental and quasi-experimental designs can produce credible impact evaluation findings, but there is a difference, and their classifications signal what that difference is. Including effect evaluation in the integrated monitoring and evaluation (M&E) plan is crucial rather than using it as a standalone component. To establish a causal relationship, impact evaluations rely on a set of experimental and quasi-experimental methods. in this design, individuals are ranked based on specific, measurable criteria. What is the purpose of the impact evaluation? Process evaluations and monitoring both provide information on how your program is running and whether it is meeting expectations. In this blog, we will walk you through the next steps in the process from understanding the core elements of an impact evaluation work plan to designing your own impact evaluation toidentify the real difference your interventions are making on the ground. As data collection should be targeted toward the mix of evidence needed to make appropriate judgments about the program or policy, it is essential to consider what success is and how the data will be processed and synthesized to answer the key evaluation questions (KEQs). Consider the conclusions of the authors of the systematic review are there trends that emerge across countries and contexts that are relevant for you? Impact evaluation is essential for understanding the impact of a particular project or program. Although we can never observe what happens for the same set of people both with and without a program, we can use different techniques of generating a comparison group to construct what likely would have happened and this process allows us to attribute causality to your program. A theory of change that narrates how activities are understood to produce a series of outputs that help achieve the ultimate intended impacts is helpful in direct causal attribution in an impact evaluation. These are also the best people to consult to determine how to solve the problem. The next step is to construct a variety of more in-depth (mid-level and lower-level) evaluation questions that specifically address each evaluating criterion. and is a tool that will guide programme planning and . Yet there is a need to put the theory into practice in a hands-on fashion for practitioners. Use the following guidelines to help design and plan your impact evaluation. While the format of an evaluation design may vary on a case by case basis, it must always include some essential elements, including: This section provides information on the background of the intervention to be evaluated. Are you interested in identifying specific problems or general problems? Each portfolio team within the Economic Justice Program was tasked with creating its own approach to learning and evaluation to address the complex challenges that it was seeking to ad. Deliverables include an inception report, a draft report and the final evaluation report but in case of complex evaluations, monthly progress reports might also be required. To show only the systematic reviews, on the left of the screen under Filter Results, click on PRODUCTS and check the Systematic Reviews box. Organizations can use a budget analysis template provided by other organizations or consult the budgets of earlier studies that were similar to estimate the resources needed for impact evaluation. This guide written by Howard White and David Raitzer (available as a free PDF download) offers guidance on the principles, methods, and practice of impact evaluation, covering experimental, quasi-experimental and regression-based designs and approaches (not non-experimental approaches such as process tracing, qualitative comparative analysis, QUIP or comparative case studies). Customer satisfaction is maintained while your earnings are hurt. By doing an impact evaluation, you might significantly boost your revenue and attract a more devoted workforce! ask about how things were and how they are now and what changes have taken place since the intervention. ADS 201 requires that each Mission and Washington OU must conduct an impact evaluation, if feasible, of any new, untested approach that is anticipated to be expanded in scale or scope through U.S. Government foreign assistance or other funding sources (i.e., a pilot intervention). Copy the citation from the references section and paste it into a search engine. The major changes made were: For example, impact evaluations may be used to improve program implementation for the next intake of participants. A ToC also identifies which aspects of the interventions should be examined, what contextual factors should be addressed, what the likely intermediate outcomes will be and how the validity of the assumptions will be tested. Conducting Implementation Research in Impact Studies of Education Interventions: A Guide for Researchers. Once you are satisfied with your monitoring system: A process evaluation can tell you whether your program is being implemented as expected, and if assumptions in your theory of change hold. Step 1. Along this continuum, the preference in USAID's Evaluation Policy is clear: For impact evaluations, experimental methods generate the strongest evidence. The scope of the evaluation must be realistic given the time and resources available. Handbook on Impact Evaluation : Quantitative Methods and Practices "This design guide, as the title suggests, starts fromthe assumption that: The audience for this guide are those who: In depth evaluation and methodological expertise is notassumed in this guide rather readers are expected to Network is a state and local effort facilitated by the federal government that helps states, utilities, and other local stakeholders take energy efficiency to scale and achieve all cost-effective energy What variations were there in the quality of implementation in different sites? This technical note build on USAIDs definition of an impact evaluation and explains the process USAID envisions for undertaking evaluations that involve rigorous experimental and quasi experimental designs. Identifying the cause is known as 'causal attribution' or 'causal inference'. The objective of impact evaluation reflects what the evaluation aims to find out. Impact evaluation will generally answer three types of questions: Each type of question can be answered through a combination of different research designs and data collection and analysis mechanisms. All in all, a ToC helps the project team to determine the best impact evaluation methods for their intervention. Alternative causal paths and major external factors influencing outcomes may also be identified in a project theory.