Black-box

Suppose you implemented an initiative based on the broad policy objective of “an apple a day in order to keep the doctor away”. This initiative, which we dubbed An Apple a Day, involved distributing seven apples a week to each participant. A representation of this initiative without ToC would simply show the initiative followed by the intended outcome of improved health:

This is what is often referred to as a black box: it describes what goes in and what comes out without information about how things are processed in between. For an evaluation, without a ToC it is impossible to know if we measured the right aspects of implementation-quality and quantity.

Despite careful evaluation, it can be impossible to interpret evaluation results correctly in the absence of a ToC. If the initiative failed to achieve significant differences between those in a treatment group and those in a control group, it might seem the policy does not work—but it might also be that it has not been implemented properly. Maybe the apples were delivered but not eaten, or maybe they were too small, or too unripe, or over ripened to work as expected. Although the evaluation might include some measures of the quality and extent of implementation, it can be difficult to know what aspects should be included unless there is a ToC.

If the results showed that the initiative seemed to have succeeded, without a ToC we might also have trouble using these results more broadly. If we do not know what elements of the policy are important, we can only copy it exactly. In contrast, if we understand the causal mechanisms by which an intervention works, we can scale to other contexts while adapting to a different setting while still achieving the intended results.

Finally, if we had mixed results, where the policy worked on only some sites or for some people, we might not even notice them if we were looking only at the average effect. If we did see differential effects in different contexts (e.g., men vs. women and rural vs. urban), an evaluation without ToC leaves us in the position of having to do simple pattern matching (e.g., using the policy for the groups or sites where it has been shown to work), but with little ability to generalize to other contexts since we don’t understand the reasons for differences between contexts.

Causal Chain

A ToC of the Apple a Day initiative would identify how we understand that the initiative works and what intermediate outcomes need to be achieved for the initiative to work. This allows us to distinguish between implementation failure (not done right) and theory failure (done right but still did not work).

You might for example, understand the causal processes that occur between delivering apples and improved health as articulated below. It shows the causal mechanisms involved in producing the two changes (changed behaviour and changed health status). An evaluation based on this ToC would collect data about each element. You could then work through the causal chain to see if it breaks down somewhere and examine if you can identify implementation failures (e.g., apples were not delivered), failure of the theory, or missing details (e.g., dosage: maybe vitamin C levels does increase from eating apples, but not enough to make a difference).

Context

Developing a ToC also helps clarify differential effects by learning from those participants for whom the initiative was effective. If the initiative works for some groups but not for others or works at some sites but not at others, it is important to try to understand why by identifying possible explanations and then checking these out empirically.

The simple ToC of the An Apple a Day initiative is based on the assumption that the apples are both necessary and sufficient—that is, the apples will lead to good health in all circumstances and without contributions from other factors. Developing a more complete ToC would focus on the differential effects we might expect for different types of participants. We would investigate whether the theory works in some contexts but not in others.

For example, the An Apple a Day initiative might only work for certain types of participants—for example, those who are affected by diseases related to inadequate nutrition. For people affected by infectious diseases, apples by themselves might not be enough to improve health.

Or data might show the initiative worked for men but not for women. Possible explanations might be differences in labour force patterns which affect access to fresh fruit or differences in nutritional needs related to pregnancy.

Other context variables such as the cultural values attached to the fruit may also come into play. For example, if apples are considered only for children and you’ve learned the mechanism through which eating apples provides vitamin C and as a result generates health benefits, then for adults you may choose to substitute apples with other food that is high in vitamin C.

Multiple Pathways

Vitamin C might not be the only pathway through which apples contribute to improved health. Social interaction with the apple deliverer or physical exercise from playing with the apples may also contribute as depicted in the diagram:

Similarly, there may be alternative pathways for how eating apples improves people’s health:

Multiple pathways may apply simultaneously. Knowing the strengths of different pathways, may allow you to identify alternatives for apples where apples are not readily available. Depending on which pathway(s) are valid, would lead to different critical features in implementation that should be ensured. For example, if health improvements come about through increased fibre consumption, eating the whole apple, not just drinking the juice, would be important.

Ecosystem: Sphere of Influence

Whether the health of participants in the Apple a Day initiative actually improves also depends on physical exercise. Although the Apple a Day initiative is not active in the field of physical exercise, sporting clubs could offer opportunities for collaboration such as education about a healthy lifestyle.

Negative Side-effects

Negative elements are visualized with a slight red background (see figure below). For the Apple a Day initiative, an undesired effect may be that, the delivery of apples causes people to visit the supermarket less frequently. As a result they purchase other fruits less, resulting in a decline in the variety of fruits the person has access to and subsequently a decline in health:

Feedback Loops

Feedback loops demonstrate the complexity and dynamism of reality. Using the case of the Apple a Day initiative once again, there could be a social interaction effect taking place. As more of the intended beneficiaries enrol, the growth in user base adds to the program’s reputation and visibility. The trust this produces again attracts more of the intended beneficiaries, which further adds to the program’s reputation and visibility.