Summary and helpful tools
Monitoring and evaluation (M&E) processes can be among the most effective ways to foster learning for sustainable capacity development. Yet, even thought there are now many innovative approaches that recognise the importance of learning in M&E, there has yet to be a significant paradigm shift towards adopting these new approaches for all capacity development initiatives.
The many good reasons to integrate learning into M&E of capacity development initiatives include: improving the capacity development process as will as recognising results; making the management of the capacity development more adaptive and responsive; being more inclusive and working to bring beneficiary and participant perspectives into consideration; contributing to organisational development and management capacity development; promoting active use of theories of capacity, its development, and change, all of which improve the quality of design and implementation of capacity development processes; using errors or failures as learning opportunities, rather than as something to be hidden or falsified; and promoting an evaluative culture in which enhanced learning, multiple accountabilities, transparency and organisational understanding of change and impact become the norm.
Some of the common characteristics of effective learning approaches to M&E are:
- Holding central the understanding that continual learning is essential for sustainable capacity change.
- Involving multiple stakeholder groups in ways that balance their interests and priorities, including accountability to participants and beneficiaries.
- Combining methods that generate both quantitative and qualitative data, which together lead to more comprehensive understanding.
- Using iterative, continual reflective feedback approaches to determine what is happening in the capacity development process and why it is happening.
However, there are some challenges to overcome when trying to integrate learning as a fundamental purpose when M&E systems are being developed. The history of M&E for donor accountability has established a culture of donor ownership of M&E and it can take time to negotiate an approach that has elements that are helpful for all stakeholders and in which all accept the validity of what others need from the system. There is still no easy way to resolve the inherent tension of trying to work with both accountability and learning in M&E processes at the same time.Additionally, as yet there are no well established and universally accepted mechanisms to bring learning from M&E of implementation into other arenas such as policy making or academic study.
There are several helpful tools available to work with learning and capturing qualitative information in M&E systems and some can be used both for qualitative and quantitative data, so they do not exclude accountability needs. Some M&E tools and processes can be capacity development tools in their own right, especially:
- Establishing learning objectives at the start of the capacity development process
- Integrating the action-reflection-learning-planning cycle into implementation activities
- Using action research as areflection tool to answer the question ‘How can I/we improve my work?‘
- Asking why? In any circumstance asking not only what happened, but why it happened will contribute a lot to generating learning from both ongoing and periodic review processes.
Introduction
Monitoring and evaluation (M&E) processes can be among the most effective ways to foster learning for sustainable capacity development. Unfortunately, because in in the past M&E has most usually framed and designed by the need for accountability learning has not been established as a primary focus of M&E systems. It is now recognised that the intended beneficiaries are often the most neglected stakeholder group because of the predominantly upward focus of accountability. This history of M&E meant that the approaches and tools used for accountability, which were often quantitative rather than qualitative, became the starting point for developing M&E processes for capacity development. While there are now many innovative approaches that recognise the importance of learning in M&E, there has yet to be a significant paradigm shift towards adopting these new approaches for all capacity development initiatives.
Benefits of learning approaches that foster M&E
There are many good reasons to integrate learning into M&E of capacity development initiatives. Some of the most frequently cited benefits are that learning approaches:
- Focus on getting the process right in additional to recognising the results. Given the long-term and complex nature of many capacity development initiatives the process orientation is necessary because it is important to know not only that something happened, but also why it happened. Generating this information and understanding creates a body of evidence about what works and what doesn’t work in any given context.
- Foster a broad learning approach to implementation that is helpful for adaptive and responsive management to guide improvement of the process, including planning more relevant and realistic next steps. This is especially important when working in complex systems, because using learning to guide implementation is more important than measuring outputs and outcomes that may not accurately reflect the total complexity of the situation.
- Tend to be more inclusive and can work well to bring beneficiary and participant perspectives into consideration, thereby leading to more comprehensive, relevant and deeper understanding, which in turn contributes towards achieving sustainable change.
- Enable understanding of what is happening by drawing on the realities of experience.
- Contribute to organisational development and management capacity building through the promotion of self-assessment, feedback, reflection and internal and external dialogue.
- Promote active use of theories of capacity, its development and of change, all of which improve the quality of design and implementation of capacity development processes.
- Use errors or failures as learning opportunities, rather than treating them as something to be hidden or falsified.
- Have proven benefits as incentives and for building confidence of participants
- Lead to enhanced understanding of context including the socio-political, cultural and power factors in the environment
- Promote multi-way and multi-level learning among stakeholder groups.
- Promote an evaluative culture in which enhanced learning, multiple accountabilities, transparency and organisational understanding of change and impact become the norm.
Characteristics of effective learning approaches to M&E
Studies have shown that there are some common characteristics among the effective learning approaches to M&E, namely they:
- Hold central the understanding that continual learning is essential for sustainable capacity change.
- Involve multiple stakeholder groups and balance their interests and priorities, including assuming that accountability to participants and beneficiaries is equally as important as accountability to the donors. The questions to be answered and the methods used are therefore based on multiple needs including the need to demystify methods so that everyone can participate and have a voice. If structured and conducted appropriately self-assessment processes contribute significantly to the overall success of any capacity development initiative.
- Combine methods to generate both quantitative and qualitative data, which leads to more comprehensive understanding. For example, the type of information that is best obtained through stories, case studies or other creative means significantly enriches, broadens, deepens and sometimes explains any data that has been gathered against predetermined indicators.
- Work with all key internal and external actors to establish dialogue about the linkages between capacity development activities and what they lead to in ways that promote understanding of all the factors that are relevant.
- Use iterative, continual reflective feedback approaches to determine what is happening in the capacity development process and why it is happening. This allows for internal recognition of challenges, successes and priorities as they emerge. Having this information available to help drive the process enhances both the quality of the capacity development practice and ownership of the capacity development process.
Become capacity development tools in their own right because of the emphasis on collaborative analysis and decision-making and recognition of what has been learned.
Challenges
Despite all the advantages there are a number of challenges to overcome when developing an M&E system with learning as one of its fundamental purposes.
- There is often an inherent tension when trying to work with both accountability and learning in M&E processes at the same time. Most notable is that the accountability focus on measurement of results can make implementers risk averse and unwilling to admit to problems, whereas a learning focus holds problems as a rich source of invaluable learning that can help to improve future implementation and practice. There is also the syndrome of ‘regressive’ learning in which implementers learn what is needed to fulfil donor requirements, and ignore all other learning opportunities arising from the work.
- The history of M&E for donor accountability has established a culture of donor ownership of M&E. Letting go of old methods and controls in order to change that culture to establish another that holds multi-stakeholder learning central to M&E will take time and might be very difficult for some stakeholders to manage.
- There are political dimensions to the choices about the purpose of M&E and how it should be conducted. It can take time to negotiate an approach that has elements that are helpful for all stakeholders. There is a fundamental challenge in getting different stakeholder groups to accept the validity of what others might want from M&E. Those who need hard facts and figures to show that money has been well spent may not readily see the use of a story that explains how someone is now doing something differently. Similarly a person whose life has changed as the result of an intervention might not even know what an Excel spread sheet is, let alone value the information it holds.
- Bringing learning from M&E of implementation into other arenas such as policy making or academic study calls for multiple horizontal and vertical learning loops that have a very different nature and purpose to the predominantly vertical structures in which most M&E systems for accountability operate. While some initiatives are already established it is going to take time for their benefits to spread and be universally accepted.
A selection of tools
The list below shows some of the tools that are currently in use for working with learning and capturing qualitative information in M&E systems. Note that some of these tools can be used for both qualitative and quantitative data, so they do not exclude accountability needs, but rather work with both together. The first four in the list are all very effective approaches that can be integrated into any capacity development process and M&E systems.
- Establish learning objectives: The most effective way for to ensure that learning is integral to M&E processes is to integrate learning into the design of the capacity development process. This can be done by developing indicators and objectives about the learning necessary for capacity to be developed and sustained i.e. focusing on what needs to be learned in order to create the product rather than on the product itself. For example a disaster management department needs constant and up to date information about relevant factors in the regional environment. Rather than setting an objective about the product needed, e.g. “By the end of X, the management of the disaster planning department will have an analysis of the regional environment”, a learning focus would set an objective about the learning skills needed, for example: “By the end of X, the management of the disaster planning department will be able continuously to scan and analyse the regional environment.”
- Action reflection learning planning cycle: this was developed to overcome the frequently observed problem of activities leading straight to the planning of more activities without any time taken to reflect on and learn from those already completed. It is a simple but very effective monitoring tool for structuring reflection and learning processes.
- Action research: Action research is a learning and change methodology now in use in many different disciplines where professional development is needed, especially education and health. It is a tool for learning by reflection and at its simplest action research starts with the question ‘How can I/we improve my work?’ Action research can be conducted by individuals and groups.
- Ask why? Even in an M&E system without a learning focus, or it integrate learning into accountability mechanisms a very simple and easily applied approach is to work with the question ‘Why?’ In addition to noting what has happened, evaluators and participants in routine monitoring activities can keep asking questions like ‘Why was this activity effective?’, ‘Why did this intervention not achieve the expected results?’ or ‘Why was this intervention more successful with group A than with group B?’ The answers to such questions will contribute a lot to generating learning from both ongoing and periodic review processes.
Other tools in current use are:
- 360 degree audits
- Appreciative inquiry: this is about searching for the best in people and their organisations. It approaches problem solving and future planning from a positive, ‘what if the best happened?’ perspective, rather than by analysing problems and their causes
- Benchmarking against well-defined indicators
- Customer satisfaction surveys
- Empowerment evaluations
- Most Significant Change: can be used to capture change that has happened during the capacity development initiative, but not directly intended or related to the initiatives goals, and then understand the linkages and causal factors.
- Organisational processes: including strategies; strategic plans; annual plans and budgets; strategic reviews; peer reviews; organisational climate reviews; annual participatory review and reflection (self-assessment); processes; annual; reports; internal governance annual review; external and internal audits; and open information policy.
- Individual performance management processes: appraisals; coaching, mentoring and supervisionOrganizational self assessments and action plans
- Outcome mapping
- Story telling (can be particularly useful as it allows all participants to recognise their part in larger change processes)
- Use of evidence in relation to best practice standards
This page has been compiled using the following resources
Hovland, I. Knowledge Management and Organisational Learning: An International Development Perspective An Annotated Bibliography (2003) ODI Working Paper 224.
ODI. Evaluating humanitarian action using the OECD-DAC criteria: An ALNAP guide for humanitarian agencies (2006) (ALNAP Active Learning Network for Accountability and Performance in Humanitarian Action)
Pearson, J. Integrating learning into organisational capacity development of Cambodian NGOs: lessons learned from the ICCO Partners Project, Forthcoming in Development in Practice
Ramalingam, B. Learning how to learn: eight lessons for impact evaluations that make a difference (2011) ODI Background Note
Taylor, P and A. Ortiz. IDRC Strategic Evaluation of Capacity Development, “Doing things better? How capacity development results help bring about change” (2008)
Watson, D. Monitoring and evaluation of capacity and capacity development. (Discussion Paper 58B). Maastricht: ECDPM
Next section: How to define and map a change process