Sign in or 
| Version | User | Scope of changes |
|---|---|---|
| Jun 13 2009, 9:45 AM EDT (current) | chachabooth | 130 words added, 20 words deleted |
| Jun 13 2009, 9:35 AM EDT | chachabooth | 1561 words added, 1 photo added |
Changes
Key: Additions Deletions
Evaluation as Learning. An introduction
a) What is evaluation?
Here are two definitions from recent literature about peacebuilding evaluations:
The term assessment and systematic and objective evoke associations to research – and that is also what evaluation is: It is research about a project, programme or policy that tries to answer specific questions by collecting and analyzing data. Just as in social science research, there are many different ways of doing evaluations. Evaluation could be characterized as action-oriented research, as it is meant to inform decisions-making.
b) Evaluation: Practitioner’s concerns and benefits
In general, evaluations are often perceived as time-intensive, consuming scarce financial and human resources, making futile attempts to quantify the unquantifiable, formulating unrealistic recommendations or diverting the attention of staff and management away from more important tasks. In the field of peacebuilding, practitioners often express the concern that established evaluation approaches don’t do justice to the nuanced and complex work of peacebuilding.[3] While these concerns are all important and have to be taken into account when planning the evaluation, the needs for and benefits from evaluation clearly outweigh the costs:
c) How to do an evaluation?
As mentioned before, there are many different ways of doing evaluation. It is more like a toolbox and the choice of tools has to be carefully considered. A guiding question for this choice can be: What do you want to learn from the evaluation?
Church and Rogers describe the steps for an evaluation in detail, as does the OECD/DAC guidance working draft. While no comprehensive outline shall be given here, the main stages are described below:[4]
Stage 1: Evaluation preparation
In order to be useful, evaluation needs to be planned during the stage of project design/project planning. Not only is a question about evaluations often a part of proposal formats, but planning the evaluation at an early stage also makes sure that costs involved in the evaluation are part of financial plans and that ownership of the evaluation is created. The project team should therefore drive preparing the evaluation, even though it is also possible and useful to hire an evaluator already at this stage, who could be involved in different stages of project design and for instance also conduct the baseline study.
During evaluation preparation a number of decisions have to be taken, staring with determining what the team wants to learn from an evaluation. This should be guided by considering what information is needed to inform decisions, improve future performance and understand how and why the intervention is contributing to positive change (or is not). Evaluation objectives can, according to Church and Rogers, be grouped into three main themes.
In a next step or in conjunction with determining the evaluation objectives, it needs to be decided, what the audience of the evaluation will be. There are two principle audiences for an evaluation, users and readers. While every evaluation has to address a user, readers are optional.
Also, it needs to be decided what type/types of evaluation will be used in the project’s/programme’s evaluation. A distinction between formative, summative and impact evaluations can be drawn. Formative evaluations (also mid-term evaluation) occur during – around the middle – project implementation; summative ones near or at the end of implementation and impact evaluations some time after a project has been finalized.
Another important decision is the choice of an evaluation approach. There is a wide range of approaches, such as action evaluation, goal-free evaluation, self-evaluation, theory-based evaluation, results-based evaluation or utilization focused evaluation.[5] The evaluation approach is the style or philosophy of the evaluation. When choosing an evaluation approach, the evaluation objectives determined earlier in the preparation process are key, i.e. the approach selected should be the one best-suited for achieving the evaluation objectives.
In addition decisions about the scope of the evaluation (geographic coverage and the degree to which the conclusions will be generalized) and who should conduct the evaluation need to be considered. Based on all these decisions, the evaluation budget can be developed.
Stage 2: Evaluation management
The process of evaluation management begins with the creation of Terms of Reference (ToR), which is a guide to the evaluation. Then an evaluation plan needs to be developed, which can be done either by the project team or the evaluator/s. The plan operationalises the guidelines given in the ToR. The evaluation plan should specify how data will be collected, timing and location of conducting the evaluation, etc.
Stage 3: Evaluation utilization: Learning and sharing
Once the evaluation has been conducted according to plan and a final report has been produced, it is sometimes assumed that the process of evaluation is finished. However, this is actually where the decisive step is taken, namely the utilization of the conclusions and recommendations from the evaluation. Utilization has two facets: 1) Internal learning and 2) External sharing. Church and Rogers[6] apply David Kolb’s experiential theory of learning[7] in order to show how evaluation and learning go together.
Learning in Evaluation: Applying Kolb's learning cycle.
Adapted from Church and Rogers, Designing for Results, p. 180.
d) A word on evaluation criteria in peacebuilding
In recent years, a lot of effort has been put in refining “traditional” evaluation criteria for development work, namely relevance, effectiveness, impact, sustainability, and efficiency, for usage in the peacebuilding domain. These criteria are not only important for the evaluators of a specific project/programme. Rather, reflecting these criteria can be a good guide during the planning stages. Some of the most important criteria are:[8]
a) What is evaluation?
Here are two definitions from recent literature about peacebuilding evaluations:
“Evaluation is the systematic acquisition and assessment of information gathered on specific questions to provide useful feedback for a program, organization or individual. (…) Evaluation is commonly known to serve two purposes: learning and accountability.”[1]
“Evaluation offers systematic and objective assessment of the relevance, effectiveness, impact, sustainability and efficiency of interventions.”[2]
The term assessment and systematic and objective evoke associations to research – and that is also what evaluation is: It is research about a project, programme or policy that tries to answer specific questions by collecting and analyzing data. Just as in social science research, there are many different ways of doing evaluations. Evaluation could be characterized as action-oriented research, as it is meant to inform decisions-making.
b) Evaluation: Practitioner’s concerns and benefits
In general, evaluations are often perceived as time-intensive, consuming scarce financial and human resources, making futile attempts to quantify the unquantifiable, formulating unrealistic recommendations or diverting the attention of staff and management away from more important tasks. In the field of peacebuilding, practitioners often express the concern that established evaluation approaches don’t do justice to the nuanced and complex work of peacebuilding.[3] While these concerns are all important and have to be taken into account when planning the evaluation, the needs for and benefits from evaluation clearly outweigh the costs:
- Evaluation is a unique opportunity for learning about a project or programme
- Evaluation can help to improve and professionalise peacebuilding work
- Evaluation reports can be shared with others and so help other actors to improve their work.
c) How to do an evaluation?
As mentioned before, there are many different ways of doing evaluation. It is more like a toolbox and the choice of tools has to be carefully considered. A guiding question for this choice can be: What do you want to learn from the evaluation?
Church and Rogers describe the steps for an evaluation in detail, as does the OECD/DAC guidance working draft. While no comprehensive outline shall be given here, the main stages are described below:[4]
Stage 1: Evaluation preparation
In order to be useful, evaluation needs to be planned during the stage of project design/project planning. Not only is a question about evaluations often a part of proposal formats, but planning the evaluation at an early stage also makes sure that costs involved in the evaluation are part of financial plans and that ownership of the evaluation is created. The project team should therefore drive preparing the evaluation, even though it is also possible and useful to hire an evaluator already at this stage, who could be involved in different stages of project design and for instance also conduct the baseline study.
During evaluation preparation a number of decisions have to be taken, staring with determining what the team wants to learn from an evaluation. This should be guided by considering what information is needed to inform decisions, improve future performance and understand how and why the intervention is contributing to positive change (or is not). Evaluation objectives can, according to Church and Rogers, be grouped into three main themes.
| Evaluation Theme | Evaluation objective |
Why and how is the agency conducting this intervention? | Appropriateness (or relevance) considerationconsideration: [IsIs the intervention the best for the situation and the desired goal? Review of theory of change included.]included. Strategic alignmentalignment: [AreAre the activities in line with the organisation’s mission and core principles? To effect the change articulated in its mission, an organization needs to allocate all its resources and attention to that change.]change. |
How well was the intervention implemented? | Management and administrationadministration: [HowHow well was the project organized and run?]run? Cost accountability Implementation process appraisalappraisal: [ExaminesExamines the quality of the conflict transformation techniques used in project implementation.]implementation. |
What were the results of the intervention and how long will they last? | Output identification Outcome identification Impact assessment Adaptability of changechange: [LooksLooks at whether the changes on outcome and impact level can adapt over time to shifts in the context and to different stresses and demands. This is a relatively new area of evaluation and needs to be further researched.]researched. |
In a next step or in conjunction with determining the evaluation objectives, it needs to be decided, what the audience of the evaluation will be. There are two principle audiences for an evaluation, users and readers. While every evaluation has to address a user, readers are optional.
Also, it needs to be decided what type/types of evaluation will be used in the project’s/programme’s evaluation. A distinction between formative, summative and impact evaluations can be drawn. Formative evaluations (also mid-term evaluation) occur during – around the middle – project implementation; summative ones near or at the end of implementation and impact evaluations some time after a project has been finalized.
Another important decision is the choice of an evaluation approach. There is a wide range of approaches, such as action evaluation, goal-free evaluation, self-evaluation, theory-based evaluation, results-based evaluation or utilization focused evaluation.[5] The evaluation approach is the style or philosophy of the evaluation. When choosing an evaluation approach, the evaluation objectives determined earlier in the preparation process are key, i.e. the approach selected should be the one best-suited for achieving the evaluation objectives.
In addition decisions about the scope of the evaluation (geographic coverage and the degree to which the conclusions will be generalized) and who should conduct the evaluation need to be considered. Based on all these decisions, the evaluation budget can be developed.
Stage 2: Evaluation management
The process of evaluation management begins with the creation of Terms of Reference (ToR), which is a guide to the evaluation. Then an evaluation plan needs to be developed, which can be done either by the project team or the evaluator/s. The plan operationalises the guidelines given in the ToR. The evaluation plan should specify how data will be collected, timing and location of conducting the evaluation, etc.
Stage 3: Evaluation utilization: Learning and sharing
Once the evaluation has been conducted according to plan and a final report has been produced, it is sometimes assumed that the process of evaluation is finished. However, this is actually where the decisive step is taken, namely the utilization of the conclusions and recommendations from the evaluation. Utilization has two facets: 1) Internal learning and 2) External sharing. Church and Rogers[6] apply David Kolb’s experiential theory of learning[7] in order to show how evaluation and learning go together.
Learning in Evaluation: Applying Kolb's learning cycle.
Adapted from Church and Rogers, Designing for Results, p. 180.
d) A word on evaluation criteria in peacebuilding
In recent years, a lot of effort has been put in refining “traditional” evaluation criteria for development work, namely relevance, effectiveness, impact, sustainability, and efficiency, for usage in the peacebuilding domain. These criteria are not only important for the evaluators of a specific project/programme. Rather, reflecting these criteria can be a good guide during the planning stages. Some of the most important criteria are:[8]
- Relevance: Is the intervention on the right track to contribute to peacebuilding? Does it respond to peacebuilding needs? Here, it is also important to see how the intervention takes what others do into consideration and how it relates to overall strategies and policy frameworks.
- Effectiveness: Has the intervention/project/programme reached its objectives (expected results)? It is important to note that besides intended effects, there might be unintended ones that can be either positive or negative.
- Impact: What positive and negative, primary and secondary, intended or unintended long-term effects did an intervention produce? How was the conflict and peacebuilding environment affected? The question of impact evaluation is a difficult one, and sometimes it is not possible to assess long-term effects – yet or because attribution is impossible. In such a situation, outcomes and theories of change should be at the heart of the evaluation and the question whether the intervention can in the long-run contribution to peace needs to be considered.
- Sustainability: This is the continuation of benefits after the intervention has been completed.
- Efficiency: Looks at how economically inputs have been translated into results.
- Coherence (and coordination): Is the intervention consistent with the larger policy context (on national and international level)? Is it coordinated with other policies, programmes or projects?
- Linkages: This criterion refers to the connections (linkages) between the intervention and activities, projects, programmes, policies on other levels.
References:
Church, Cheyanne/Rogers, Mark M., Designing for Results. Integrating Monitoring and Evaluation in Conflict Transformation Programs (Chapters 7-11), Search for Common Ground/United States Institute of Peace/Alliance for Peacebuilding, 2006. <http://www.sfcg.org/programmes/ilr/ilt_manualpage.html> (accessed 14 April 2009).
Organisation for Economic Cooperation and Development, Development Assistance Committee (OECD/DAC), Guidance on evaluating conflict prevention and peacebuilding activities (Working draft for application period), OECD 2008. <http://www.oecd.org/secure/pdfDocument/0,2834,en_21571361_34047972_39774574_1_1_1_1,00.pdf > (accessed 14 April 2009)
Lederach, John Paul/Neufeldt, Reina/Culbertson, Hal, Reflective Peacebuilding. A Planning, Monitoring, and Learning Toolkit (Chapter 12: Evaluation as Learning), Joan B. Kroc Institute for International Peace Studies and Catholic Relief Services East Asia Regional Office, 2007, pp. 63-66. <http://kroc.nd.edu/sites/default/file/reflective_peacebuilding.pdf> (accessed 14 April 2009).
Notes:
Church, Cheyanne/Rogers, Mark M., Designing for Results. Integrating Monitoring and Evaluation in Conflict Transformation Programs (Chapters 7-11), Search for Common Ground/United States Institute of Peace/Alliance for Peacebuilding, 2006. <http://www.sfcg.org/programmes/ilr/ilt_manualpage.html> (accessed 14 April 2009).
Organisation for Economic Cooperation and Development, Development Assistance Committee (OECD/DAC), Guidance on evaluating conflict prevention and peacebuilding activities (Working draft for application period), OECD 2008. <http://www.oecd.org/secure/pdfDocument/0,2834,en_21571361_34047972_39774574_1_1_1_1,00.pdf > (accessed 14 April 2009)
Lederach, John Paul/Neufeldt, Reina/Culbertson, Hal, Reflective Peacebuilding. A Planning, Monitoring, and Learning Toolkit (Chapter 12: Evaluation as Learning), Joan B. Kroc Institute for International Peace Studies and Catholic Relief Services East Asia Regional Office, 2007, pp. 63-66. <http://kroc.nd.edu/sites/default/file/reflective_peacebuilding.pdf> (accessed 14 April 2009).
Notes:
[2] OECD/DAC, Guidance on evaluating conflict prevention and peacebuilding activities. Working draft for application period, p. 13.
[3] OECD/DAC, Guidance on evaluating conflict prevention and peacebuilding activities. Working draft for application period, p. 13.
[4] The following largely draws on Church and Rogers, Designing for Results, pp. 92ff. The OECD/DAC guidance lists the following steps: Planning and preparing the evaluation, conducting the evaluation, concluding and learning from the evaluation. The DAC guidance seems more directed towards evaluators and is less detailed about the planning and preparation phase.
[5] While these approaches shall not be explained in detail here, there are very good summaries in Church and Rogers on pp. 114-121 and the OECD/DAC guidance on pp. 85-89.
[7] See here Kolb, David A., Experiential Learning: Experience as the Source of Learning and Development. Prentice-Hall, Inc., Englewood Cliffs, N.J. 1984.
