Summary and action points
To set up a capacity measurement system, for any level or context, it is necessary to have in place: a clearly articulated framework for defining capacity; a definition of the starting point; a vision of where the process is trying to get to; a theory of change that guides intervention planning; indicators for key points in the process; and criteria to provide a framework for measurement.
Action steps
- Decide who needs to be involved and how. All capacity development processes have many stakeholders, some who are involved from start to finish; others who only participate in part of the process. There will be a constantly changing group whose needs have to be accommodated. It is important to guard against approaches that only answer the needs of only one type of stakeholder. Measurements need to be meaningful to all actors identified as relevant to the process. There are many ways to involve stakeholders appropriately.
- Decide what needs to be measured and the criteria for measurement. There is no single, universally agreed set of capacity development dimensions to use in a measurement system. Measurement dimensions need to be specific to the context and cover all levels – individual, organisational, sectoral and institutional, types of capacity – hard and soft, and the themes for application in any capacity development framework being used. Some frameworks use the OECD DAC monitoring and evaluation criteria of relevance, effectiveness, efficiency, ownership, impact and sustainability as a way to structure how they approach measurement. A balanced approach ensures that all relevant aspects of capacity are covered at one time or another. Important points to remember are to:
- Ensure the system is capable of capturing not only predicted but also unexpected results
- Guard against too heavy a focus on hard capacity
- Focus on the aspects and dimensions of capacity that are worth measuring and that you will be able to produce either qualitative or quantitative evidence for what you have chosen.
- Create a measurement framework to fit the context. Creating a specific framework can ensure that the measurement process and tools fit the capacity development process. The dimensions and criteria discussed above can be used to identify a starting point, which could be: inputs and outputs, outcomes, or impact. Using an iterative approach i.e. deciding where to start and doing the details one step at a time (rather than trying to map out the whole thing at the start) allows for effective response to what is emerging and any changes in the environment.
- Test the framework. The framework needs to be first tested, and later reviewed regularly, for relevance and practicality before it is put to extensive use.
- Select tools. When tools are being selected it should be remembered that: all tools should be adapted to local context and needs; all tools have advantages and disadvantages according to context, and this should be taken into account when using them; and, a mix is needed to cover all the different measurement requirements i.e. different tools will be needed at different stages in the process. Some tools to consider are: outcome mapping; stories of change; most significant change; case studies; random sampling; tracer studies; ladder of change; theory-based evaluation; rapid appraisal methods; cost-benefit and cost-effectiveness analysis; Logical Framework; and public expenditure tracking surveys. NOTE: client satisfaction is an important area of measurement that is often left out of monitoring and evaluation systems.
It is important to remember that the measurement system should be integral to the design and implementation of the capacity development initiative from the start. This can help to foster both effective use of the theory of change and reflective learning practices.
Introduction
Measuring is usually determined in two different ways: monitoring, which is the ongoing measurement of an intervention process in action as it happens; and evaluation, which is a periodic review, usually covering a broader range of criteria than everyday monitoring processes.
In order to set up an effective capacity measurement system, for any level or context, it is necessary to have some other things in place, namely: a clearly articulated framework for defining capacity; a definition of the starting point; a vision of where the process is trying to get to; a theory of change that guides intervention planning; indicators for key points in the process; and criteria to provide a framework for measurement.
Who needs to be involved in measurement and how?
All capacity development processes have many stakeholders. Some organisations and individuals will be involved from start to finish, others will only participate in part of the process. This means that there is a constantly changing group who need to understand what is happening and whose interests need to be accommodated. Target participants, implementing agencies, government institutions, the public, civil society and donors may all have different ideas about what capacity is needed and how it can or should be developed. While it may not be appropriate or possible to respond to all expressed needs and interests, wherever possible the system used should incorporate steps and tools that involve the key stakeholders as much as possible. It is particularly important to guard against approaches that only answer the needs of only one type of stakeholder. Measurements need to be meaningful to all actors identified as relevant to the process. This is especially important when working on long-term, incremental implementation approaches that can only be effective through multi-stakeholder engagement. If the assessments of progress made, challenges encountered, etc. do not have legitimacy in the eyes of all key stakeholders they are unlikely to engage with ongoing steps in the process.
Steps to involve key stakeholder groups appropriately include:
- Initial definition of current capacity and future capacity needs
- Identification of indicators
- Agreement about the measurement framework and tools to be applied, and how the different stakeholders will be involved
- Use of participatory tools when implementing the framework
- Feedback and discussion on measurement findings before they are finalised
- Regular consultation and reviews to ensure that both long-term and newer stakeholders are all kept up to date with the purpose, process and results of the measurement activities
Deciding what needs to be measured and the criteria for measurement
There is no single, universally agreed set of capacity development dimensions to include in a measurement system. As with all other aspects of capacity development, measurement dimensions need to be specific to the context. In order to be fully comprehensive, the measurement system needs to cover all the relevant levels, types of capacity, and the themes for application in any capacity development framework being used. It may not be necessary to pay attention to all components all the time but, if the measurement is to be holistic and meaningful, there should be a balanced approach that ensures that all relevant aspects of capacity are covered at one time or another. Taking a holistic approach is also more likely to answer the needs of multiple stakeholders. Some dimensions that can be considered when framing what needs to be measured are:
- The goal and objectives of the capacity development process
- Impact, outcomes and outputs, and the linkages between them
- Hard capacities (products, services, results) and soft capacities (learning, adaptation, relationships, etc.)
- Product and process of the capacity development intervention
- Competencies of individuals, collective capabilities of groups, and overall system capacity to achieve a development goal
- Perspectives of participants, providers, managers, donors, etc. and especially the customer perspective of service users
- Performance alone, or together with other capabilities as per the ECDPM model
- Pre-determined criteria AND openness to unexpected results
This last point is really important because the complex nature of capacity development in many contexts means that unexpected results, which can sometimes be more informative and interesting than what was expected, will always emerge.
Discussing which of these dimensions to include and or focus on when developing the measurement framework is a good way to surface the needs and understanding of different stakeholders and then bring them together to create a balanced approach to measurement. It is important to guard against too heavy a focus on hard capacity as it is now understood that such a focus can be detrimental to the development of holistic capacity. What matters is to focus on the aspects and dimensions of capacity that are worth measuring and for which either qualitative or quantitative evidence can be obtained.
Many models and frameworks use the OECD DAC monitoring and evaluation criteria as a way to structure how they approach measurement. This could be applied to capacity development as follows:
- Relevance of the capacity development intervention
- Effectiveness of the capacity development intervention
- Efficiency of the capacity development intervention
- Ownership of the capacity development intervention
- Impact of the capacity development intervention
- Sustainability of the capacity development intervention
Again, not all of these criteria will be relevant all the time, they should be considered as and when appropriate according to the stage of the implementation process.
Creating a measurement framework to fit the context
The components of capacity being considered and the stage of the capacity development process both need to be taken into consideration in order to organise the measurement process and tools and create a framework specific to the context.The framework needs to take account of what needs to be measured and the criteria for measurement, as discussed above, but that doesn’t help to decide where to start. Thinking about an iterative approach can be useful, which means deciding where to start in terms of the overall theory of change then doing the details one step at a time. This way allows for more appropriate responses to what is emerging, and for taking account of any changing factors in the environment.
One way of choosing where to start is:
- Inputs and outputs: This approach starts with the activities and tries to track the effect and impact they created. The advantage of this approach is that it addresses issues of attribution, and covers the quality of the capacity building activities and interventions. Disadvantages are that it will tend to focus on specifics rather than cumulative interventions, and cannot easily be used to produce holistic measurement.
- Outcomes: This is the zoom in, zoom out approach to measurement. It takes a recognised change as the starting point then zooms in to see what might have caused the change, and zooms out to see what impact the change is having. The advantage of this approach is that it can embrace multiple capacity development initiatives and other factors relevant to change. The disadvantage is that it may not be possible to identify the effectiveness of specific activities.
- Impact: This method starts where there is identifiable change in terms of the development goal and then works backwards to see what has caused that change to happen. This method is helpful for technical capacity development, where there is an easily defined end-product. It is likely to show how improved capacity within different organisations can together contribute to wider changes at society or community levels.
It is not necessary to make either/or choices, because at different times and for measuring different aspects of the process, many methods will be relevant, individually or together.
Any framework created needs to be tested for relevance and practicality before it is put to extensive use, it should also be reviewed regularly to keep it up to date with developments in the overall process.
It is important to remember that the measurement system should be integral to the design and implementation of the capacity development initiative from the start. This can help to foster both effective use of the theory of change and reflective learning practices.
Some tools that can be used for measurement
- Outcome mapping: this approach looks at the network of players involved in a capacity development process over the long term and looks at the changes in their behaviour in terms of a spread of outcomes.
- Stories of change: working with stories is a good way to capture the experiences of participants in capacity development processes that are too complex to be measured in other ways. To avoid this approach being criticised as anecdotal it should be used in conjunction with other tools.
- Most significant change: is a system of recording and analysing changes that were not anticipated or predicted at the start of the intervention. As with stories of change the data from this method needs to be triangulated by other tools.
- Case study: a case study can provide a very rich, full and analytical assessment of capacity changes. The criteria for the study and analysis need to be clearly stated at the start. Case studies take time so this is not a tool that can be used for large measurement needs.
- Random sampling: Choosing a selection ofindividuals or organisations to be the subject of case studies can allow for extrapolation of findings to build up a more comprehensive picture of the full impact of a capacity development process.
- Tracer study: this is a longitudinal study that does regular measurement exercises, qualitative or quantitative, over time in order to be able to track progress against pre-determined criteria.
- Ladder of change. A hypothetical ladder is developed by key stakeholders, starting with a statement of the current situation on the middle rung. Statements of how the situation might get better or worse are put above and below on the ladder. The exercise is repeatedly regularly in order to provide comparative information about changes.
- Client satisfaction: there are many different ways to measure client satisfaction and how it changes over time – citizen report cards, client satisfaction surveys, focus group discussions, and so on.
- Theory-Based Evaluation: this is a process of examining all the factors identified in relation to the theory of change or capacity articulated as underpinning the intervention process. Depending on the theory in use, this approach has the benefit of being able to deal with complexity and of avoiding the cause and effect constraints of the Logical Framework approach.
- Rapid Appraisal Methods: activities such as individual interviews, focus group discussions, observation, mini-surveys, etc. can be used as quick and relatively low-cost ways to get information. These approaches will not provide comprehensive information unless triangulated.
- Logical Framework: standard practices for planning applied to capacity development activities. This approach has many critics as being too dependent on cause and effect logic and too constrained by project timeframes, neither of which are always appropriate for capacity issues.
- Cost-Benefit and Cost-Effectiveness Analysis: these tools can be used to assess whether or not activities and resources used for a capacity development process can be justified by the results. The danger with these tools for capacity development is that they measure everything in monetary values.
- Public Expenditure Tracking Surveys: these surveys track the flow of public funds and the extent to which resources reach target groups. This type of survey is helpful for capacity measurement at institutional level, but again can constrain measurement to monetary considerations.
When selecting tools the important things to remember are that:
- All tools should be adapted to local context and needs
- All tools have advantages and disadvantages for different contexts, which should be taken into consideration before they are put into use
- A mix is needed to cover all the different measurement requirements and different tools will be needed at different stages in the process
This page is drawn from the following resources
Engel, Paul, Tony Land and Niels Keijzer, A balanced approach to monitoring and evaluating capacity and performance, ECDPM 2007
IDRC, IDRC-supported Capacity Building: Developing a Framework for Capturing Capacity Changes
Ortiz Alfredo and Peter Taylor, Learning purposefully in capacity development: Why, What And When To Measure? UNESCO & IIEP, 2009
Simister, N., Monitoring and Evaluating Capacity Building: Is it really that difficult?, INTRAC Praxis Paper 2, 2010
UNDP, Measuring capacity, 2010
Watson, D, “Measuring capacity development” in Capacity Development in Practice Ubels et al: Earthscan, London, 2010.
World Bank, Monitoring and Evaluation: Some Tools, Methods & Approaches, 2004
Next section: How to develop M&E processes that foster learning