Description
The Progress theme is used to report on progress made and results achieved on the elements of the ToC. This theme brings to light “if and where” a ToC works out differently than expected, thus enabling learning.
Your ToC forms the basis for monitoring at output level, monitoring and evaluation at outcome and impact levels and for learning. Data forms the basis for this and the Progress theme is the place to present this data and analyses. It requires clarity about the ToC elements you need or want to follow and why, before you think of Key Performance Indicators.
Using Key Performance Indicators (KPI’s) of quantitative and qualitative nature with data like tables, diagrams, illustrative stories from the field, etc., allows social organisations to stay informed about the results and progress of the intervention. In sum, it is the Progress theme where measured results, their analysis and interpretation find their way back into Changeroo.
Celebrating Progress and Signalling Risk
To engage stakeholders and generate enthusiasm, it is important to celebrate success as you make progress toward your vision of success. Equally, it is important to foster a learning environment and to signal the areas of your ToC that warrant attention and monitoring because they run the risk to bring the change process to a halt. Such risk may for example come from:
- Lack of progress
- Uncertainty about an assumption
- Decreased commitment from essential staff or partner
- Changes in basic conditions for success
- Lack of experience in an area so that you want to keep a close eye on how it goes
- Interdependencies within your ToC that cause an entire ToC to collapse from a small error in a specific ToC element
To celebrate success and signal risk associated with a specific part of your ToC, Changeroo allows you to set a traffic light for the ToC elements/relationships. This traffic light functions as a compass and steers (management) attention. You can set this traffic light at the following levels:
- Level 1: success, on track, achieved, low-risk
- Level 2: slightly off track, some KPI’s may be on track while others aren’t, moderate attention needed, moderate risk
- Level 3: off track, vital attention required, high risk
Stakeholder Engagement
Social organisations
You can inform stakeholders about the progress being made: what is going well and where results are stalling and where the ToC may need revision. You can also inform stakeholders about how you monitor the change process.
Stakeholders might engage in several ways, such as:
- Identifying key information needs
- Defining performance/learning questions
- Suggesting indicators/ KPI’s and ways to collect and analyse information
- Participating in the sense-making of the information collected for the KPI’s.
- Providing feedback on the KPI’s used and their measurement method
- Offering their experiences on how the organisation performs
- Suggesting available benchmarks
Stakeholders
As a stakeholder, this theme allows you to learn about the progress being made and how the change process gets monitored. You see what is going well and where results are stalling and where the ToC may need revision.
You can engage in several ways, such as:
- Supporting the identification of elements of the ToC that need to be followed.
- Supporting the definition of the performance or learning questions, keeping in mind to focus on what you truly need to know to take decisions. You are encouraged to use the discussion page to share your needs, performance or learning questions.
- Critically examining the KPI’s used by the organisation and their measurement methods.
- Suggesting (alternative) KPI’s. But bearing in mind there are always trade-offs to be made between the benefits and costs of measurement. You are encouraged to use the discussion page first to discuss any of such ideas.
- Knowing of any available benchmarks the organisation may benefit from, you could also suggest those.
- Supporting the interpretation and analysis of data.
Questions to ask
- What elements of the ToC do we need to follow to collect the data needed to take decisions?
- What do we need to know about a ToC element to understand its contribution to the vision of success?
What do we need to see happen in our context to understand the extent to which we’ve contributed to change? - Which ToC elements within each pathway of our ToC need to be measured to prove our rationale behind that pathway?
- What ToC elements and assumptions are at risk and should we monitor? What exactly is it that is at risk?
- For each assumption in our ToC: what may happen if that assumption proves wrong?
- Are the indicators chosen SMART enough to measure progress? Please look below for more information about quality of indicators.
- Have we included both leading and lagging indicators for each pathway of our ToC?
- For outcome measures, the SROI Guide suggests asking stakeholders how they know that change has happened to them. “For example, if the outcome was an increase in self-confidence, ask the people whose self-confidence is increased what they now do as a result, or ask them to tell you what increase in self-confidence means to them.”
What to Describe
It should be noted that each indicator is preceded by the performance or learning questions it relates to.
Typical information to include for an indicator, either quantitative or qualitative:
- Indicator: the actual variable being measured, including clear description, threshold and timeline (if relevant). A threshold represents a minimum performance (e.g., a $12 per hour job for at least six months) with the indicator measuring the percentage of the population who perform within the threshold. The timeline delineates a timeframe when the performance of the threshold needs to be achieved (e.g., within two months of graduation).
- Objectives and timelines: the targets set out for the future and the timelines set out to achieve them.
- Population: the group that you are measuring. This specifies the scope or boundaries of what is included in the indicator. It can for example refer to a specific group of people, in a geographic area, targeted in a specific time frame.
- Methodology (data collection and analysis): information about how you will collect the data, frequency, sample size, how you will measure and other relevant information about data collection and data analysis.
- Responsibility: who will be responsible for the collection and analysis of data?
- Critical reflection and interpretation of data
- Results of the indicator: This may also include benchmarks if available and relevant. Whereas quantitative indicators will use figures to show results, qualitative data might use short narrative text
- Argumentation: argumentation for including the indicator (and not others) and the chosen measurement method(s).
- For outcomes and impact: the extent to which changes can be attributed to your activities and outputs. To prevent over claiming and to make your indicators credible, you could seek to determine how much of the outcome would have happened anyway and what proportion of the outcome is attributable to your interventions. This requires you to control for trends that affect the KPI irrespective of your activities as well as other activities outside your organisation’s control. See the SROI Guide for more information. Changeroo also discusses it under the “Validation & assumptions” theme.
Thinking About Progress
Elements’ Attributes
The first priority should be to clearly detail the attributes of each ToC element, and to deal with their measurement later. After all, ToC implementation is to focus on the desired outcomes and their desired features, and not just those that are (easily) measurable.
Choosing What to Measure: Key Information Needs and Performance/Learning Questions
Then, you will need to make choices about what aspects of your ToC to follow, based on your key information needs. For example, you might be interested about whether a change has taken place. But you might also be interested to test whether the assumption you made is correct or whether a necessary condition for the success of your intervention is still at play. These key information needs will be translated into performance/learning questions. Focusing on key information needs and performance/learning questions helps avoid preparing a long list of indicators. It also enables you to critically look at complex issues where it is difficult to come up with few fixed indicator.
Reasons for Measurement Include:
- Stakeholders expect to learn about certain aspects of the ToC or progress on these aspects is what gets them enthusiastic.
- To measure the realization of your vision of success. Sustainable systemic change often takes time before it takes place (e.g. improved farmers livelihood in a low income country). Therefore, it is essential to measure intermediate results towards systemic change and to include a range of indicators that cover the change process (e.g. income generated from crops, increase in yield, whether farmers are practicing good agricultural practices taught in training and through extension).
- To monitor and learn about the validity of specific steps within your pathways of change. If the ToC does not work out as expected, measurement informs you where your ToC broke down. This allows you to make targeted adjustments as needed instead of having to conclude that your entire ToC isn’t working. ToC elements involved in steps that are crucial to the success of your ToC and/or whose validity are topic of discussion are best to be measured.
- To test and learn about assumptions
- To report to your funder (accountability purposes)
It is also essential to think beforehand on how you or stakeholders will use the information generated. This helps to keep focus and provide the information in a way that the end users can actually use it efficiently.
Deciding How to Measure: Choosing Indicators
Various criteria can be applied for indicator selection. Commonly used criteria are:
- Relevant and Specific
The indicator should be clearly related to areas in which your organisation is expected to make some difference. Factors and measures that are largely subject to external influences should be avoided. The relevance to your organisation should be clear, together with the reasons for wanting to monitor the change or factor. - Credible
There must be a reasonable case for the view that changes in the selected indicators are related, directly or indirectly, to the intervention. - Unambiguous
The indicator should be clearly defined, so that measurement and interpretation is unambiguous. For example: in ‘(improved) access to […] services’, the notion ‘access’ has different aspects (such as physical, financial, geographical, gender, class or cultural barriers): what will be monitored and measured? - Consistent
Ideally, the same indicators should be measured over a long period in order to track long-term processes. However, if an indicator is not relevant anymore, if the context, priorities or objectives have changed, or there are important unexpected effects, it might be necessary to revise or replace the indicator. - Sensitive
This means that there is a short reaction time to change: the quicker results lead to change in the indicator, the more useful it is for monitoring. For example, the outcome of elections that are held once every 5 years is not a very sensitive indicator for changes in the political force field. - Feasible to collect
An important selection criterion is whether it is feasible to collect information on the indicator(s) within a reasonable time and at a reasonable cost. Monitoring of information that is too difficult, time-consuming or costly to collect, will in practice quickly be dropped. - SMART
A case is often made for SMART indicators.http://www.lltcorp.com/content/kpi-s-m-r-t-rule- Specific: unambiguous and clear in what it exactly measures
- Measurable: in terms of capturing the performance on the metric as well as the recourse required
- Achievable and realistic
- Relevant to the achievement of our vision of success and to the needs of stakeholders
- Time phrased: its value is expressed in time as for a KPI to have meaning, one needs to know the time dimension in which it is realized
Focusing on SMART indicators, however, can also have drawbacks. They can generate lots of discussion and can distract attention from the actual ToC. Also when setting goals such as “at least 80% of course participants will obtain a job within 1 month of graduation”, may not be very meaningful for complex ToCs with emergent, difficult to predict, processes and outcomes. Lastly, there is a higher risk of defining performance measures in terms of what is easy to count rather than what will be useful and meaningful and on measures or indicators of quantity without reference to quality. As a result, SMART measures often focus on completed activities (outputs) rather than achievements (outcomes).S.C. Funnell & P.J. Rogers (2011) Purposeful program theory: Effective use of theories of change and logic models. San Francisco: Jossey-Bass.
More About Indicators
Standardized Indicators
When selecting your KPI’s you might want to choose standardized indicators over custom ones, if available. Because standardized indicators are used throughout the impact community, they’ll allow for benchmarking. Two sources where you can find often-used indicators are:
- IRIS
- The Sustainable Development Goals (SDGs) by the United Nations
- Worldwide Governance Indicators (WGI) project
Different Types of IndicatorsKusters, C.S.L. and Batjes, K. with Wigboldus, S., Brouwers, J. and Baguma, S.D. (2017) ‘Managing for Sustainable Development Impact: An Integrated Approach to Planning, Monitoring and Evaluation’, Wageningen: Wageningen Centre for Development Innovation, Wageningen University & Research, and Rugby, UK: Practical Action Publishing.
Different types of indicators can be distinguished:
- Quantitative indicators: measures of quantity, such as the number of women users in a forest user committee or the percentage of women trained in income generating skills resulting in income actually increasing.
- Qualitative indicators: Descriptive; people’s judgements and perceptions about a subject (e.g. perceptions about the initiative’s impact); explain the ‘why’ behind numbers.
- Direct indicators measure directly the performance for that, which you want measured.
- Indirect or proxy indicators are generally used when direct indicators are not possible. They measure the performance on something that is believed to correlate with that which you want measured. Such a measure thus, gives an indication of the performance of what you want to measure.
- Objective indicators are factually true and not prone to subjective interpretation.
- Subjective indicators are a personal opinion. Such indicators often come in the form of people’s self-reported judgements and perceptions.
It is good practice to mix subjective and objective indicators that complement each other. Subjective self-reported indicators provide more flexibility in what to measure but are also less reliable than objective indicators.
Examples of Different Types of Indicators
| Quantitative | Qualitative | |
| Information |
|
|
| Methods | Direct observation (measuring/counting) | Discussion groups with villagers about how quality of life has changed |
| Analysis & Reporting |
|
|
| Types of indicator | Examples |
| Simple quantitative indicators |
|
| Complex quantitative indicators |
|
| Compound indicators |
|
| Proxy indicators |
|
| Qualitative indicators – open ended |
|
| Qualitative indicators – focused |
|
| Quantitative | Qualitative | |
| Objective | How much have your revenues increased in the last 12 months? | A respondent responding to a question with “Yes, I own a fruit stand” |
| Subjective | Rating on a scale of 1 to 5 (fully disagree – fully agree) of the extent to which a respondent agrees with the statement “The program has changed my life” | A story from a beneficiary about how they feel a program has changed his/her life |
Leading and Lagging Indicators
A distinction can also be made between leading and lagging indicators. Leading indicators come early in the change process. Leading indicators often take the form of measuring inputs, outputs and early outcomes. They change quickly and are a precursor to the direction something is going. They thus inform early on about the likely impact and give an idea about whether activities are working.
Leading indicators are however less accurate or definitive about the extent to which the change process is taking place. For this you need lagging indicators, which are backward-focused and measure change as it has taken place. They therefore come later in the whole change process but are more accurate and definitive. Lagging indicators are the measures for the final outcomes and impact.
Closing the Loop with Outcome Measurement and Impact Measurement
Your ToC provides the foundation for the development of your impact measurement framework. There is a wide range of methodologies for impact measurement and for monitoring and evaluation. What links them all is the importance of having a ToC that lays out the expected story in advance of the changes happening. This then provides a blueprint for determining what to measure, collecting the evidence, checking other possible explanations, and learning and adjustment (Isabel Vogel).
Your ToC provides a framework for monitoring, evaluation and learning. The ToC is where impact measurement starts as it makes clear what needs to be measured to track progress as well as to validate your rationale. Therefore, it makes sense to include KPI’s in your ToC and to feed back the measurement results into your Changeroo account. This provides a basis for learning about what works and why. It fuels excitement among stakeholders as it shows you are capable of delivering the potential impact that the ToC holds as well as your intentions to learn and improve further.
Community Guidelines
To make sure the Changeroo platform is a powerful vehicle for change, we’ve put together a few ground rules.
Content pages focus on consensus, integration of information and objective facts, instead of repetition and endless texts. Social organisations can open up these pages to stakeholder feedback and input.
Guidelines
- Share your expertise and resources to be of influence and help social organisations move their social mission forward. Seek opportunities to contribute that align with your own values. Present yourself through your contributions.
- Learn from stakeholders’ feedback and improve your understanding of an organisation.
- Be constructive in your criticism.
- Be open to disagreement and listen to other people’s opinions. Change happens when people with different backgrounds and perspectives can participate in a conversation.
- Be reasonable, kind, respectful and use common sense.
Do not use hate speech, do not impersonate others, do not violate others’ privacy, do not bully, do not be unnecessarily graphic, do not spam and do not break the law (which includes respecting copyrights in your posts). - Update your ToC with developments. When you’re candid with stakeholders and tell them what we do and don’t know and make them part of the process, it’s much more engaging for them.