Measuring Impact

Most charities don’t actually measure their effectiveness; at best they report on their outputs. For example, a charity might tell you that they supply two additional teachers to a school in Africa. That sounds like a good project but perhaps it isn’t. It might encourage a cash restrained government to reallocate some of the existing teachers they supply to other schools, or perhaps their more urgent need is better infrastructure. The primary objective in this case is not to increase local employment but to improve the education of these children, so that’s what really needs to be measured. This is called Impact Measurement.

Impact measurement is a technique that has evolved rapidly over the last decade. However, it is still in a formative phase and only used by a small minority of charities. There are number of reasons for this:

• In many sectors impact is hard to measure, in some it is impossible
• The relevant time period can often be years
• Other factors may have contributed to the impact

Despite these issues, impact measurement is gradually becoming accepted as the most reliable basis for evaluating interventions.

In an ideal scenario you will be able to see a report produced by a respected independent organisation. Ideally this should include the results of a Randomised Control Trial (RCT) that will compare results of two cohorts of beneficiaries, one with and one without the intervention. However, these are still a rarity. In all cases what you should be trying to identify are the outcomes from the programme, not the outputs.

For an example of a detailed cost/benefit analysis look at this one produced by GiveWell.

Is there a Theory of Change?

Theory of Change (ToC) is a relatively new concept that helps to define how a programme or intervention will achieve the desired result. It involves a careful examination of each step in the process and of the evidence supporting its effectiveness. It can be a useful indicator of how well an organisation has considered the implications of its activities. The absence of one may indicate a lack of rigour in the organisation.

A ToC usually starts by considering:

• the Outcomes that you are seeking and these should be quite specific,
• the Outputs that will be needed to achieve this,
• the Inputs required to make the outputs happen

Let’s assume that the main objective is to improve the life chances of children by ensuring they are better educated. Following a consultation with all of the stakeholders (especially the locals) you would list all of the factors that could contribute to the objective, such as:

• increasing the number of pupils attending, especially in the final years
• improving the quality of teaching
• improving the facilities

That would lead to an action plan of Inputs, which might look like this:

• build three new classrooms over the next four years,
• pay for additional teachers commencing in two years,
• build a dormitory for girls in the second year
• collect rain water to supply freshwater,
• Install solar panels and satellite aerial to provide internet access
• plant trees to provide shade.

From this plan, budgeted costs can be established together with key indicators of progress. In the short run those indicators are likely to be mainly the outputs but as times goes by they will be replaced by outcomes such as final year pupil numbers and exam results.

If you are interested in learning more about Theory of Change there is a considerable amount of useful content at: Theoryofchange.org.