Sign in or 
- EasyEdit
- Edit tags
-
(what's this?What are these tools?
People just like you can add or edit the content on this site. If you want to try editing, but aren't ready to add to this site, try our demo area.
Read more about editing pages at Wetpaint Central.
) - Report page
Share this
Monitoring as learning. An introduction
Why monitoring?
Monitoring plays a central part in the management of peacebuilding/conflict transformation programmes. Without monitoring, one is often unable to track changes and measure the progress that it makes in building the capacities of its partners or on the changes that the work with partners produces in the conflict context.
A good monitoring system helps to inform day-to-day decision making and helps to ensure that decisions and activities are moving towards expected results. At the same time monitoring also helps to ensure that well informed changes can be made to programme design, including the expected results, and budget in case the implementation of the programme no longer leads to the desired results. The working environment in peacebuilding is one of rapid changes, capacity to manoeuvre and manage change is therefore very important both in order to stay relevant, not become counter-productive and in order to not endanger staff or partners. A good monitoring system is essential to adapt to a shifting environment to grow and develop. Monitoring also provides important information for mid-term and final evaluations. Brought together, continuous monitoring and evaluation lay the foundation for a fruitful learning process and a higher degree of accountability towards its beneficiaries and donors.
What is monitoring and how does it differ from evaluation?
Monitoring is the continuous process of gathering information about programme implementation and using this information in decision-making. It takes place as the poject/programme is running. Monitoring is different from evaluation with regards to when it takes place, what information is gathered and what decisions are informed (see table below). Nevertheless, monitoring and evaluation are closely connected. Monitoring lays the basis for evaluation.
Table: Monitoring and Evaluation compared
Adapted from Church and Rogers, p. 83.
Forms of monitoring
Monitoring can take on different lenses, i.e. track changes in different dimensions. The three most important forms - context, implementation and assumptions monitoring - are described below.
a) Context monitoring
Conflict transformation/peacebuilding interventions take place in dynamic and shifting environments. Monitoring the context – or conflict analysis – enables to take pro-active decisions as well as to safeguard the security of your staff, partners and participants. Conflict analyses should for this purpose be continuously updated. The focus should be here on actors and escalating and de-escalating factors. Also, the international context of the conflict should be monitored closely. Key informants that have access to a wide range of information are vital for updating conflict analyses. This could be local civil society actors (partners), journalists, local leaders, academics, etc. Informal meetings among programme staff and key informants can be a very good opportunity to make sense of recent developments and inform programme implementation. Based on the updated conflict analysis, the following questions can help to guide management decisions:
This form of monitoring looks at how the implementation of the project/programme is running. Thereby it is possible to see whether the project/programme is making progress. Often this is done by comparing planned activities with the activities that are carried out and by looking at the outputs of these activities. However, implementation monitoring can go beyond this and look also at higher-level results. The indicators for these higher level results are useful here. According to Church and Rogers, there are two ways to measure progress even if the results are only to be achieved by the end of project/programme implementation: 1) Ration the magnitude of change: Maybe some progress will be made towards a higher-level result, while it will only be fully achieved at the end of a programme/project. 2) Monitor steps within the process: You may want to monitor steps within a longer process.
c) Monitoring assumptions/theories of change
This form of monitoring is asking the question whether the project is having the desired influence, is leading to the change in the conflict anticipated in the theory of change? Here, we need to look at our theories of change and discuss them in light of implementation and context monitoring.
These three forms of monitoring can be seen as interlinked with each other and also as placed within three different control spheres as seen as in the figure below. In this figure all the mentioned aspects have an effect on each other, at the same time as the degrees of influence lessen the further out from the centre you get. The implementing organisation will be able to monitor all of the dimensions, but will have the biggest control and influence on how it chooses to change its implementation of activities and its theories of change in response to that monitoring.
Figure: Forms of monitoring and level of control/influence. Adapted from Lederach et. al., p. 58.
How to do monitoring?
Doing monitoring is based on the steps described earlier throughout this resource kit, namely: Theories of Change, results chains, indicators and baseline study. The results chain, together with your indicators, can be used as a guide for monitoring, in particular the implementation monitoring. If you include theories of change in your monitoring, this may help you to keep track on this dimension as well.
The data that you collected in the baseline study establishes a point of comparison. It is good to plan how you want to monitor your project/programme from the beginning. A monitoring plan can help to structure the process. Below, find an example framework that could be used for monitoring:
(Implementation) monitoring plan
Adapted from Church/Rogers, p. 91.
With this plan, the collection of data can be organised. This is, however, only the first step. Once the data has been collected, it needs to be interpreted and incorporated in decision-making processes. Also, this table does not include context monitoring or monitoring of assumptions. A good tool for monitoring can also be programme team reflection/monitoring sessions, facilitated and prepared by one staff member assigned for the PMEL process. This can also be a platform to discuss monitoring data and look at its implications for programme implementation. Besides looking at quantitative indicators these sessions could be prepared by asking all team members to use self-assessment tools or alternatively by asking them to identify the most significant changes that have taken place within the programme. Such sessions can easily also be conducted with partners and techniques for both self-assessment and for identifying most significant changes are described in the next section.
Practical tools for monitoring and data collection
Once you have developed a monitoring plan, you also need to decide on what tools you will use to collect the data that you need to learn (in order to inform decision-making). The most suitable tools will to a large extent depend on the context you are working in, the resources that you have available to do monitoring and whether you are mainly working on your own or together as a team.
Below we will present some different tools for data collection that can be very useful when monitoring.
Questionnaires
As long as the data that you want to collect is quantitative, tangible and easy to measure (rather than qualitative), a simple questionnaire asking your target group a number of questions is an easy way to do monitoring of activities. When you construct the questionnaire you need to remember to code or structure it so that you easily can summarise the results when doing data analysis. When using questionnaires for the evaluation of activities – such as workshops – it is also important to remember to hand them out to participants as soon as possible after the activity in order for participants not to forget what they thought or what the activity was about. If you use questionnaires to monitor processes it might, however, not be as important to distribute questionnaires directly after activities have taken place. In fact there can be reasons for why you would want to wait some time. Remember though that the monitoring of processes might demand the collection of more qualitative data than what a questionnaire easily allows you to collect. A good idea could then be to combine the questionnaire with other tools for qualitative data collection.
While the monitoring of indicators is important, a too narrow focus on pre-defined quantitative indicators does not guarantee that we learn from monitoring. The measurement of quantitative indicators often needs to be complemented with more qualitative forms of information gathering in order to capture the complex dimensions of capacity building and conflict transformation. Monitoring systems can be designed in a way that enables the measuring of results through both indicators and more “qualitative” forms of monitoring. This can be done through asking for the most significant change, self-assessment tools and reflection sessions
Asking for the most significant change[1]
In order to monitor different processes in contexts where the information you need is of a more qualitative nature or where people might not be used to filling in questionnaires a good tool for monitoring can be to ask for stories about the most significant change. Define a time frame and an area in which you want to monitor change and then ask your target group to describe to you the most significant change that has taken place during this time period and within the preset area. You can also ask them how this change happened if you want to specifically monitor your own assumptions about change. By asking these relatively open questions, the people you are interviewing are free to answer without being bound to your own indicators or guiding questions which allows you to see also those changes that you had not planned for as well as discover new risks within the programme in time. This method for monitoring also allows for stories of most significant change to be reported through other means than written text (e.g. images, drawings, taped interviews, videos and films) which might help in target groups that are illiterate or more used to oral traditions or who simply are not trained in formal report writing. You and your team can then sit together and analyse what changes were in line with your own visions of success, theory of change etc. Are there connections between your theory of change and how the target group describes how change happened, or do you need to adapt your theories of change to better fit with how the target group deals with and see change as occurring?
Self assessment tools
Self assessment can be a very useful tool for internal monitoring of programmes and change processes with partners as internal staff within a programme or a partner often tend to be more critical to their work than are external evaluators. One way of doing self–assessment in a structured manner is to let staff or partners fill in a results journal on a regular basis[2].
To track changes over time for example for each partner organisation you engage with, it seems useful to complete one results journal per partner. The results journal lists the indicators and asks for an assessment what the level of change with regards to indicators was (if there was any change). The level of change ranges from low (0-40 %), medium (40-80%) to high (80-100%), i.e. to what extent has the indicator been achieved. The results journal, in addition, encourages describing and reflecting on the change that has taken place and also asks for unexpected changes – that could easily be forgotten when just looking at pre-formulated results and indicators. In a last step, lessons learnt and suggestions for changes in the programme implementation can be noted. The results journal is completed by individual staff members (this can of course be extended to partners’ staff) and then discussed in common monitoring sessions taking place regularly (see below).
Results Journal
Adapted from Outcome Mapping's Outcome Journal. See Sarah Earl/Fred Carden/Terry Smutylo, Outcome Mapping. Building Learning and Reflection into Development Programmes, IDRC, 2001. <http://www.idrc.ca/en/ev-28389-201-1-DO_TOPIC.html> (accessed 29 May 2009)
Regular programme team reflection sessions
If you are working in a team a good tool for monitoring can be regular programme team reflection sessions, facilitated and prepared by one staff member assigned for the PME&L process. Such meetings can also be a platform to discuss monitoring data, do assumptions monitoring and look at implications for programme implementation. Besides interpreting data collected for quantitative indicators, these sessions could be prepared by asking all team members to use self-assessment tools. If you have collected change stories, these sessions provide the opportunity to review and reflect them. Such sessions can easily also be conducted with partners.
Remember: A sustainable monitoring system should be: simple to grasp, light to implement, provide useful information, help meeting reporting requirements.
So what you can do to make sure we learn more from monitoring?
References:
Church, Cheyanne/Rogers, Mark M., Designing for Results. Integrating Monitoring and Evaluation in Conflict Transformation Programs (Chapter 6), Search for Common Ground/United States Institute of Peace/Alliance for Peacebuilding, 2006. <http://www.sfcg.org/programmes/ilr/ilt_manualpage.html> (accessed 14 April 2009).
Rick Davies/Jess Dart, The 'Most Significant Change' (MSC) Technique. A Guide to Its Use, April 2005. <http://www.mande.co.uk/docs/MSCGuide.pdf> (accessed 8 May 2009).
Lederach, J. P., et. al. , Reflective Peacebuilding: A Planning, Monitoring, And Learning Toolkit, Notre Dame, The Joan B. Kroc Institute for International Peace Studies, University of Notre Dame and Catholic Relief Service South East Asia Regional Office 2007, pp. 57-62. <http://kroc.nd.edu/sites/default/files/reflective_peacebuilding.pdf> (last accessed 26/03/2009)
Monitoring plays a central part in the management of peacebuilding/conflict transformation programmes. Without monitoring, one is often unable to track changes and measure the progress that it makes in building the capacities of its partners or on the changes that the work with partners produces in the conflict context.
A good monitoring system helps to inform day-to-day decision making and helps to ensure that decisions and activities are moving towards expected results. At the same time monitoring also helps to ensure that well informed changes can be made to programme design, including the expected results, and budget in case the implementation of the programme no longer leads to the desired results. The working environment in peacebuilding is one of rapid changes, capacity to manoeuvre and manage change is therefore very important both in order to stay relevant, not become counter-productive and in order to not endanger staff or partners. A good monitoring system is essential to adapt to a shifting environment to grow and develop. Monitoring also provides important information for mid-term and final evaluations. Brought together, continuous monitoring and evaluation lay the foundation for a fruitful learning process and a higher degree of accountability towards its beneficiaries and donors.
What is monitoring and how does it differ from evaluation?
Monitoring is the continuous process of gathering information about programme implementation and using this information in decision-making. It takes place as the poject/programme is running. Monitoring is different from evaluation with regards to when it takes place, what information is gathered and what decisions are informed (see table below). Nevertheless, monitoring and evaluation are closely connected. Monitoring lays the basis for evaluation.
Table: Monitoring and Evaluation compared
| Monitoring | Evaluation | |
| Implemented throughout the programme or project: continuously, frequently, periodically | When? | An event, implemented at certain stages of programme implementation |
| Informs day-to-day decision making; enables management decisions throughout implementation LEARNING | Purpose | Informs future programming. Deepens understanding, why and how things have happened. LEARNING |
| Tracks progress towards results achievement; observes changes in the context | Focus | Looks at the overall picture. What has happened and why? Determines relevance, efficiency, effectiveness, impact etc. |
| Usually carried out internally, by programme staff as well as partners and participants | Who does it? | External consultants or programme staff, together with partners, participants. |
Forms of monitoring
Monitoring can take on different lenses, i.e. track changes in different dimensions. The three most important forms - context, implementation and assumptions monitoring - are described below.
a) Context monitoring
Conflict transformation/peacebuilding interventions take place in dynamic and shifting environments. Monitoring the context – or conflict analysis – enables to take pro-active decisions as well as to safeguard the security of your staff, partners and participants. Conflict analyses should for this purpose be continuously updated. The focus should be here on actors and escalating and de-escalating factors. Also, the international context of the conflict should be monitored closely. Key informants that have access to a wide range of information are vital for updating conflict analyses. This could be local civil society actors (partners), journalists, local leaders, academics, etc. Informal meetings among programme staff and key informants can be a very good opportunity to make sense of recent developments and inform programme implementation. Based on the updated conflict analysis, the following questions can help to guide management decisions:
- Are our assumptions about the context still valid? Do we have to change them?
- Are our interventions still strategic?
- Have new opportunities or constraints for our engagement emerged?
- If the conflict context has changed considerably, why are we still doing the same things?
This form of monitoring looks at how the implementation of the project/programme is running. Thereby it is possible to see whether the project/programme is making progress. Often this is done by comparing planned activities with the activities that are carried out and by looking at the outputs of these activities. However, implementation monitoring can go beyond this and look also at higher-level results. The indicators for these higher level results are useful here. According to Church and Rogers, there are two ways to measure progress even if the results are only to be achieved by the end of project/programme implementation: 1) Ration the magnitude of change: Maybe some progress will be made towards a higher-level result, while it will only be fully achieved at the end of a programme/project. 2) Monitor steps within the process: You may want to monitor steps within a longer process.
c) Monitoring assumptions/theories of change
This form of monitoring is asking the question whether the project is having the desired influence, is leading to the change in the conflict anticipated in the theory of change? Here, we need to look at our theories of change and discuss them in light of implementation and context monitoring.
These three forms of monitoring can be seen as interlinked with each other and also as placed within three different control spheres as seen as in the figure below. In this figure all the mentioned aspects have an effect on each other, at the same time as the degrees of influence lessen the further out from the centre you get. The implementing organisation will be able to monitor all of the dimensions, but will have the biggest control and influence on how it chooses to change its implementation of activities and its theories of change in response to that monitoring.
Figure: Forms of monitoring and level of control/influence. Adapted from Lederach et. al., p. 58.
How to do monitoring?
Doing monitoring is based on the steps described earlier throughout this resource kit, namely: Theories of Change, results chains, indicators and baseline study. The results chain, together with your indicators, can be used as a guide for monitoring, in particular the implementation monitoring. If you include theories of change in your monitoring, this may help you to keep track on this dimension as well.
The data that you collected in the baseline study establishes a point of comparison. It is good to plan how you want to monitor your project/programme from the beginning. A monitoring plan can help to structure the process. Below, find an example framework that could be used for monitoring:
(Implementation) monitoring plan
| Monitoring focus: Result (on different levels: outcome, output) | Indicators | Means of Verification/Information source/Data collection tool | Frequency of data collection | Who collects data? |
With this plan, the collection of data can be organised. This is, however, only the first step. Once the data has been collected, it needs to be interpreted and incorporated in decision-making processes. Also, this table does not include context monitoring or monitoring of assumptions. A good tool for monitoring can also be programme team reflection/monitoring sessions, facilitated and prepared by one staff member assigned for the PMEL process. This can also be a platform to discuss monitoring data and look at its implications for programme implementation. Besides looking at quantitative indicators these sessions could be prepared by asking all team members to use self-assessment tools or alternatively by asking them to identify the most significant changes that have taken place within the programme. Such sessions can easily also be conducted with partners and techniques for both self-assessment and for identifying most significant changes are described in the next section.
Practical tools for monitoring and data collection
Once you have developed a monitoring plan, you also need to decide on what tools you will use to collect the data that you need to learn (in order to inform decision-making). The most suitable tools will to a large extent depend on the context you are working in, the resources that you have available to do monitoring and whether you are mainly working on your own or together as a team.
Below we will present some different tools for data collection that can be very useful when monitoring.
Questionnaires
As long as the data that you want to collect is quantitative, tangible and easy to measure (rather than qualitative), a simple questionnaire asking your target group a number of questions is an easy way to do monitoring of activities. When you construct the questionnaire you need to remember to code or structure it so that you easily can summarise the results when doing data analysis. When using questionnaires for the evaluation of activities – such as workshops – it is also important to remember to hand them out to participants as soon as possible after the activity in order for participants not to forget what they thought or what the activity was about. If you use questionnaires to monitor processes it might, however, not be as important to distribute questionnaires directly after activities have taken place. In fact there can be reasons for why you would want to wait some time. Remember though that the monitoring of processes might demand the collection of more qualitative data than what a questionnaire easily allows you to collect. A good idea could then be to combine the questionnaire with other tools for qualitative data collection.
While the monitoring of indicators is important, a too narrow focus on pre-defined quantitative indicators does not guarantee that we learn from monitoring. The measurement of quantitative indicators often needs to be complemented with more qualitative forms of information gathering in order to capture the complex dimensions of capacity building and conflict transformation. Monitoring systems can be designed in a way that enables the measuring of results through both indicators and more “qualitative” forms of monitoring. This can be done through asking for the most significant change, self-assessment tools and reflection sessions
Asking for the most significant change[1]
In order to monitor different processes in contexts where the information you need is of a more qualitative nature or where people might not be used to filling in questionnaires a good tool for monitoring can be to ask for stories about the most significant change. Define a time frame and an area in which you want to monitor change and then ask your target group to describe to you the most significant change that has taken place during this time period and within the preset area. You can also ask them how this change happened if you want to specifically monitor your own assumptions about change. By asking these relatively open questions, the people you are interviewing are free to answer without being bound to your own indicators or guiding questions which allows you to see also those changes that you had not planned for as well as discover new risks within the programme in time. This method for monitoring also allows for stories of most significant change to be reported through other means than written text (e.g. images, drawings, taped interviews, videos and films) which might help in target groups that are illiterate or more used to oral traditions or who simply are not trained in formal report writing. You and your team can then sit together and analyse what changes were in line with your own visions of success, theory of change etc. Are there connections between your theory of change and how the target group describes how change happened, or do you need to adapt your theories of change to better fit with how the target group deals with and see change as occurring?
Self assessment tools
Self assessment can be a very useful tool for internal monitoring of programmes and change processes with partners as internal staff within a programme or a partner often tend to be more critical to their work than are external evaluators. One way of doing self–assessment in a structured manner is to let staff or partners fill in a results journal on a regular basis[2].
To track changes over time for example for each partner organisation you engage with, it seems useful to complete one results journal per partner. The results journal lists the indicators and asks for an assessment what the level of change with regards to indicators was (if there was any change). The level of change ranges from low (0-40 %), medium (40-80%) to high (80-100%), i.e. to what extent has the indicator been achieved. The results journal, in addition, encourages describing and reflecting on the change that has taken place and also asks for unexpected changes – that could easily be forgotten when just looking at pre-formulated results and indicators. In a last step, lessons learnt and suggestions for changes in the programme implementation can be noted. The results journal is completed by individual staff members (this can of course be extended to partners’ staff) and then discussed in common monitoring sessions taking place regularly (see below).
Results Journal
| Monitoring period | ||
| Contributors to this monitoring update | ||
| Vision of success in the programme: | ||
| Outcomes | Indicators | Level of change |
| Low (L) Medium (M) High (H) LMH LMH LMH LMH | ||
| Outputs | Indicators | Level of change |
| LMH LMH LMH LMH | ||
| Description of the change: What happened? How did change happen? Why? | ||
| Who or what contributed to the change? | ||
| Sources of evidence | ||
| Unanticipated change | ||
| Lessons/ required programme changes/reactions | ||
Regular programme team reflection sessions
If you are working in a team a good tool for monitoring can be regular programme team reflection sessions, facilitated and prepared by one staff member assigned for the PME&L process. Such meetings can also be a platform to discuss monitoring data, do assumptions monitoring and look at implications for programme implementation. Besides interpreting data collected for quantitative indicators, these sessions could be prepared by asking all team members to use self-assessment tools. If you have collected change stories, these sessions provide the opportunity to review and reflect them. Such sessions can easily also be conducted with partners.
Remember: A sustainable monitoring system should be: simple to grasp, light to implement, provide useful information, help meeting reporting requirements.
So what you can do to make sure we learn more from monitoring?
- A basic principle for learning from monitoring is to ask questions and to confront yourself with possibly uncomfortable perspectives!
- Make sure that monitoring is not an isolated activity, but is inbuilt in your working processes. This can be done by regular team monitoring sessions, prepared and facilitated by one staff member, where decision-makers are present.
- Be creative in your monitoring: While the results chain and indicators are the key element of doing monitoring the implementation, reflect on theories of change and use other tools to collect more qualitative data, such as significant change stories.
[1] Inspired by “The most Significant Change Technique” developed by Rick Davies for CCDB see R. Davies and J. Dart 2005 ”The most significant change MSC Technique, A guide to its use”
[2] The described results journal is adapted from Outcome Mapping's ”Outcome Journal” to better fit the PME&L system as it has been developed in this resource kit so far. For more information about Outcome Mapping please see Sarah Earl/Fred Carden/Terry Smutylo, Outcome Mapping. Building Learning and Reflection into Development Programmes, IDRC, 2001. <http://www.idrc.ca/en/ev-9330-201-1-DO_TOPIC.html> (accessed 29 May 2009); Sarah Earl/Fred Carden/Terry Smutylo, Brochure on Outcome Mapping. The Challenges of Assessing Development Impact, International Development Research Center (IDRC), 2001. <http://www.idrc.ca/uploads/user-S/10945665201om_pamplet_final.ppt> (accessed 29 May 2009)
References:
Church, Cheyanne/Rogers, Mark M., Designing for Results. Integrating Monitoring and Evaluation in Conflict Transformation Programs (Chapter 6), Search for Common Ground/United States Institute of Peace/Alliance for Peacebuilding, 2006. <http://www.sfcg.org/programmes/ilr/ilt_manualpage.html> (accessed 14 April 2009).
Rick Davies/Jess Dart, The 'Most Significant Change' (MSC) Technique. A Guide to Its Use, April 2005. <http://www.mande.co.uk/docs/MSCGuide.pdf> (accessed 8 May 2009).
Lederach, J. P., et. al. , Reflective Peacebuilding: A Planning, Monitoring, And Learning Toolkit, Notre Dame, The Joan B. Kroc Institute for International Peace Studies, University of Notre Dame and Catholic Relief Service South East Asia Regional Office 2007, pp. 57-62. <http://kroc.nd.edu/sites/default/files/reflective_peacebuilding.pdf> (last accessed 26/03/2009)
chachabooth |
Latest page update: made by chachabooth
, Aug 26 2009, 2:38 AM EDT
(about this update
About This Update
Edited by chachabooth
1 word added 14 words deleted view changes - complete history) |
|
Keyword tags:
context monitoring
implementation monitoring
learning
monitoring plan
monitoring vs. evaluation
(edit keyword tags)
More Info: links to this page
|
There are no threads for this page.
Be the first to start a new thread.
