How to formulate capacity indicators for different contexts and levels

Summary and key action points

It often takes a long time and complicated process for the original intention to develop capacity for achievement of a development goal to materialise in the form of the impact achieved as a result of the capacity having been developed. Indicators are useful for keeping track of what is happening as the process unfolds.  Working with capacity development indicators has many acknowledged benefits, but also some tough challenges. Innovative practices are now providing some answers to the challenges and also providing ways to meet multiple needs. 

Action steps

  • Identification of stakeholders who need to be involved in defining and using the indicators.  Effective capacity indicators reflect the interests and choices of all key stakeholders, especially participants in the capacity development activities. Stakeholders need to develop the indicators through dialogue about the critical functions and performance needed to achieve the development goal, and the best way to measure steps of progress.
  • Gathering baseline information. An indicator of progress or improvement can only be fully meaningful if the starting point has been established according to clear criteria. The most useful form of baseline for capacity indicators is a mapping exercise of current performance, perhaps a description of the quantity and quality of the services produced by an organisation or sector.  The baseline can be simple, something ‘good enough’ that everyone can understand and use as their starting point.The mapping should cover soft capacity issues like leadership, power relations, learning and so on. Additionally information is needed about the relevant factors in the institutional environment, especially any that are recognised to be blocks to capacity development.
  • Decision about the types of indicators needed. Indicators for hard and soft capacities, and for the different levels, will be very different because the capacities themselves are so different. Similarly the indicators for the capacity development process and the capacity product it has been designed to develop will also be different.
    • Hard capacities are generally easier to measure by the quantity and quality of outputs but may not be a good indicator of overall capacity. A focus only on indicators of hard or formal results will not, in the long run, be very helpful for providing information about the development of sustainable capacity. For this type of capacity applying SMART criteria can be helpful.
    • Soft capacities are very important indicators of overall system capacity and the potential for sustainable change. Even if they cannot be measured, they can often be observed.
  • Sequencing the indicators over time. In long-term capacity development processes, especially those in complex contexts, it is not helpful, and maybe not even possible, to formulate indicators for all stages of the process at the start. As capacity emerges individuals and organisations will be able to focus on higher level and more sophisticated targets for their own performance.  Sequencing also allows for flexible experimentation and adaptation.  Ideas for better indicators and ways to measure will also often emerge as the process progresses.

The ‘capacity for what?’ question should be kept central to all thinking about indicators.

Introduction

There is often a long process between the original intention to develop capacity for achievement of a development goal and the impact that is ultimately achieved as a result of the capacity.  From start to finish many things will happen and many factors will be influential.  Indicators are useful for keeping track of what is happening as the process unfolds.  There is still a lot of debate and experimentation about the best way to develop and use indicators for monitoring and evaluating capacity development. Some of the issues being explored are: What is being assessed – process, product or both? Who needs the indicators? How can they use them to best effect? What values are at work in the measurement of capacity? How can soft capacities be measured? Innovative practices are now providing some answers to these questions and also providing ways to meet multiple needs. 

Why are indicators needed?

Capacity development indicators have the acknowledged benefits of:

  • Describing both the desired future state (results) and the process to reach it
  • Supporting monitoring and evaluation of progress at all levels
  • Giving an impetus to all stakeholders to clarify what they mean about capacity in the specific context
  • Providing information for key stakeholders
  • Being part of the of capacity development itself process itself when used appropriately in reflection and learning exercises
  • Generating a sense of progress and success which is, in turn, an incentive to keep going

Among the main challenges are:

  • How to formulate indicators for soft capacity that cannot easily be defined in quantitative formats
  • Understanding what is worth measuring as opposed to what can be measured, and,
  • Framing indicators that are simple and meaningful for all stakeholders when working in complex needs and situations

Who should define and use the indicators?

If they are to be effective capacity indicators must reflect the interests and choices of all key stakeholders, especially participants in the capacity development activities. Thus involving those stakeholders in the formulation of indicators can be critical to their engagement. It is often found that one of the difficulties with capacity indicators is that they tend to be oriented to donor and project Logical Framework needs and do not necessarily come from an agreement between all relevant parties about current capacity or future needs. It is, therefore, really important that the key stakeholders develop the indicators through dialogue about the critical functions and performance needed to achieve the development goal, and the best way to measure steps of progress. Given that capacity development processes often continue over the life of several projects it is helpful to have in place a process not only to assess and report on indicators, but also regularly to re-establish local ownership and commitment to working with them.

Starting point – baseline information

An indicator of progress or improvement can only be fully meaningful if the starting point has been established according to clear criteria.  It is only possible to know the progress made if conditions at the start were made clear. Too often in the past this step has been overlooked. The starting point is often referred to as the baseline, which can take a number of forms, some of which can be very complicated.  The most useful form of baseline for capacity indicators is a simple ‘mapping’ exercise of current performance, perhaps a description of the quantity and quality of the services produced by an organisation or sector. Focusing the baseline on current capacities and outcomes is a way to avoid monitoring inputs or gaps rather than achievements and progress. The mapping should cover soft capacity issues like leadership, power relations, learning and so on. Additionally, information is needed about the relevant factors in the institutional environment, especially any that are recognised to be blocks to capacity development. Overcoming blocks to change might be a good indicator of an effective capacity development process.

As noted above, if indicators are to be useful for all stakeholders, they all need to be involved in baseline assessments in order to reach agreement about the current situation and what improvements would look like.  This might need time in order that some participants in the process can reach an appropriate level of self-awareness of their current situation. Another good reason to involve all stakeholders is that they probably already have a lot of the necessary information, or will be key sources for getting it.

Types of indicators

The first difference to be clear about is between indicators for the process and for the capacity (product) it has been designed to develop. For example the capacity development intervention might focus on organisational learning, so the process indicator would be the integration of learning tools into everyday work processes.  The product indicator might be that the learning practices have led to better problem solving and improved quality of service for customers.

Some approaches to planning, monitoring and evaluation use SMART criteria (specific, measurable, achievable, results oriented and time bounded) for indicators, as well as , or sometimes instead of applying these criteria to objectives. As mentioned in other sections, there is no right or wrong in these choices, each of which have to be made according to what will be most appropriate for the context and needs.

Hard capacities can generally be easier to measure, for example by the fact that something now exists, or by the quantity and quality of outputs. For this type of indicator the SMART criteria can be very helpful. However, the existence of a hard capacity, for example a legislative or policy framework, is often not such a good indicator of overall capacity as are the soft capacities needed to create or implement the framework. This is because a new law might be an indicator of the skills of a legal department in a government institution, whereas of much more interest to the development goal of good governance will be the process through which the government consulted and negotiated with relevant citizens groups, civil society the private sector and internally about formulation of the law. A focus only on indicators of hard or formal results will not, in the long run, be very helpful for providing information about the development of sustainable capacity because firstly, they do not reflect behaviour and attitudes and, secondly, they can keep the focus of monitoring so narrow that important improvements in soft capacity are missed.

Some capacities are observable, even if they are not measurable.  Things like the quality of interactions between institutions, the confidence with which a woman addresses a public meeting or the ability of a leadership team to think strategically, while not easy or perhaps even impossible to capture in quantitative terms are nevertheless important indicators of capacity and the potential for change being sustained. Some soft capacities can be hard to measure because they relate to personal behaviour and attitudes and the individuals concerned may find the assessment intrusive or threatening.  This is another reason to ensure their engagement and participation in formulating the indicators right from the start. Interpretation of the data and judgements about any indicators will often be subjective, especially so with soft capacities. 

Sequence the indicators over time

In long-term capacity development processes, especially those in complex contexts, it is not helpful, and maybe not even possible, to formulate indicators for all stages of the process at the start. As different aspects of the capacity development process are implemented there will be many outputs and outcomes. There will also be changes in the institutional environment that while not directly linked to the capacity development process, will have a direct impact on implementation and results.  All of these factors need to be reviewed periodically against existing indicators and when necessary new indicators should be formulated.  Additionally, as individuals or organisations develop different capacities, they will be able to focus on higher level and more sophisticated targets for their own performance. 

The final reason for sequencing is to enable flexible experimentation and adaptation.  Given the use of capacity indicators is a relatively new discipline, no one can know for sure at the start of a process if the chosen indicators are going to be the most helpful and informative. Maybe the wrong things are being measured, or the right things are being measured in the wrong way.  Ideas for better indicators and ways to assess them will often emerge as the process progresses.

Selected examples from the literature

From Capacity, Change and Performance, ECPDM Study 2008

A local government project in the Philippines identified a key factor in selecting local government units (LGUs) as partners was their ‘readiness’ or ‘receptivity’ to work with an external intervention. The checklist of indicators of readiness or absorptive capability that the programme developed over time, was:

  • The willingness of mayors to support capacity interventions;
  • The level of interconnections between each LGU and others involved in the reform;
  • The level of internal teamwork between the mayor and the elected council;
  • The level of community involvement and participation in LGU affairs;
  • The state of the security situation in the area; and
  • The LGU’s perception of the relevance of the external assistance on offer.

From The design and use of CD indicators, Peter Morgan 1998

WHOSE CAPACITY?

CRITICAL FUNCTION

(CAPACITY TO DO WHAT?)

INDICATORS

Local financial officials, district assembly members, central financial officials, political authorities at all levels

Decentralizing payment functions from line ministries to local governments

Ability of the system to transfer funds between authority levels (say within 45 days of the end of the quarter) and or produce audited statements within six months of the end of the fiscal year

Operational staff at the field level of certain central agencies and ministries

Need to coordinate information amongst six ministries working on environmental issue of soil erosion in a particular region

25% increase in the number of projects that require contributions from two or more departments

From Defining and measuring capacity results, UNDP Capacity Group

Human Resources

  1. The institution has adequate staff in all key positions.
  2. Compensation is adequate and equitable.
  3. Monetary and non-monetary incentives support targeted behaviour.
  4. The staff turnover rate is low.
  5. Opportunities exist for staff professional development and on-the-job training.
  6. Staff is held accountable for getting work done according to clear performance standards.

Management

  1. Institutional management has a high degree of autonomy.
  2. The institution's management style is participatory and enabling.
  3. Managers have a clear sense of realistic goals and priorities.
  4. There is effective delegation of management responsibility to second-level managers.
  5. Managers have a high level of fiscal and operational awareness.
  6. Staff can clearly describe their roles and responsibilities.

Enabling Environmental Mastery

  1. Appropriate links exist with other institutions.
  2. Bureaucratic support is evident for the institution's activities.
  3. Major environmental influences are identified and assessed for relative degree of influence and are accurately forecast.
  4. The institution has controlled access to essential natural resources and other inputs.
  5. The institution has access to needed technologies.

From Capacity Development Results Framework, World Bank Institute 2009

Indicators

Description of indicators

Commitment of leaders to the development goal (DG)

Social and political leaders consistently and frequently make statements or take leadership actions and decisions supporting the DG.

Stakeholder participation in decisions about the DG

Decision-making processes about the DG consider all stakeholder opinions, and government and other organs of the state are responsive to the views of civil society and the private sector.

Stakeholder voice in decisions about the DG

Stakeholders know their rights related to the DG, claim those rights, and communicate their grievances and proposals for change to the government and legislature.

Transparency of information to stakeholders about the DG

Government and other public service entities provide accurate, relevant, verifiable, and timely information about the DG and explain actions concerning the DG in terms that stakeholders and other stakeholders can use to make decisions

Legitimacy of the policy instrument

Processes for decisions about policy instrument are informed, transparent, participatory, and deliberate. Policy instrument is perceived as desirable and appropriate within the local system of norms, values, beliefs, and definitions. The actions and sanctions prescribed by the policy are perceived as fair by stakeholders. Rights to appeal are assured.

From the Royal Government of Cambodia: National Capacity Development Framework for the Three-year Implementation Plan (IP3) of the National Program for Sub-National Democratic Development (2010)

a. CD Outputs: Immediate results produced by CD interventions or processes. CD products produced or CD services provided. Specific indicators to be measured are (but not limited to) are:

Individual Level:

  • Number of CD activities organized, number of people involved
  • Issues, concerns clarified through assessments, facilitation or reflection
  • Measure the extent of self-awareness developed through CD interventions (coaching)

Organizational Level:

  • Number and type of outputs produced (e.g. structures, plans, systems, proposals, reports etc.)
  • Extent and type of organizational issues/problems clarified
  • Identification of alternative options/solutions identified

b. CD Outcomes: Changes in people’s behaviours and organizational practices that resulted from the use of CD outputs. Specific indicators to be measured (but not limited to) are:

Individual level:

  • Staff applying their learning into their workplace
  • Changed behaviour, practices, beliefs, perceptions as a result of the intervention
  • Staff follow standards of good practices

Organizational level: 

  • Better leadership
  • Adoption of new plans, systems, structure, roles/responsibility, policies, and practiced / adhered to them accordingly
  • Application of best management practices (decision-making, meeting,monitoring and evaluation, etc.)  

c. CD Impact: Changes in staff performance; changes in organizational functioning, environment and performance. Specific indicators to be measured are:

Individual level:

  • Staff work more productively
  • Improved work quality of staff
  • Staff get recognitions from leaders

Organizational level:

  • Functions effectively and efficiently to achieve mandates proactively and able to adapt to externalchanging environment
  • Citizen express satisfaction with services provided
  • Organisation receives good cooperation from development partners and relevant stakeholders
  • Staff in the organization are well motivated, committed and have high moral
  • Organizations have developed relationship/cooperation with other public sector organizations, NGOs, IOs, CBOs, and the private sector to deliver governance
  • Staff have developed better relationships, working environment, teamwork and collaboration to deliver mandates as one.
n/a