A lesson from Rwanda: Don't confuse scale with impact

Ah, scale. Foundations, nonprofits, anti-poverty programs all pursue scale. Advice on how to scale abounds, in reports and articles like Getting to Scale, Strategies to Scale Up Social Programs and Three Things Every Growing Nonprofit Needs to Scale.

But scale is not impact. Indeed, there’s often tension between the two. “If you have $1 million to spend, do you want to target 1,000 people or 100 people?” asks Andrew Zeitlin, a development economist who teaches at Georgetown.

Unintended consequences arise when governments or nonprofits try to serve too many people with too few dollars, as Zeitlin and Craig McIntosh of UC San Diego found when they did a study in Rwanda comparing a conventional USAID nutrition and sanitation program with a program that simply gives people cash, with no strings attached. Their study is the first to be released as part of a bold new cash benchmarking initiative at USAID that I wrote about last week in the New York Times, under the headline Is Cash Better for Poor People Than Conventional Foreign Aid?

Some coverage of their study, in Vox and Quartz, said, at least in the headlines, that “cash won,” and while that’s arguably true, the study’s findings were more complicated: It found that neither the nutrition program nor the equivalent dollars given away in cash did much good, but that a larger amount of cash had significant impact.

A better takeaway from the study would be that “everybody won” because USAID, the nonprofits running the programs, economist and the rest of us, are learning more about what works and what doesn’t in global development.

Unsurprisingly, one thing that doesn’t work is spreading too few dollars over too many beneficiaries. The nutrition and sanitation program, called Gikuriro and run by Catholic Relief Services, cost about $140 per household, and involved a mishmash of offerings in agriculture, health education, sanitation, savings and lending, all aimed at improving child nutrition and health. The CRS website says:

Parents who participate in the Gikuriro are happy and proud to be able to prepare healthy food using their own means. They are confident that their family members are no longer at risk of malnutrition because of what they’ve learned about creating balanced diets.

Unfortunately, that confidence was misplaced. About the Gikuriro program, the economists Zeitlin and McIntosh write:

We uncover little evidence that directly confirms the program’s theory of change in driving primary child outcomes...No consistent impacts appear for consumption and wealth outcomes, or for health knowledge and sanitation practices.

In plain language, that means that the $19m allocated to the five-year Gikuriro program is mostly going to waste.

The equivalent amount of cash, distributed by nonprofit GiveDirectly, didn’t do any better in terms of improving the nutrition or health of children. It did help families drive down debt, which is no small thing.

The Gikuriro program helped people save more money, through village savings groups. But Zeitlin told me: “Spending $140 per household, over the course of a year, no matter how you do it, doesn’t seem to move the needle on the primary outcomes.”

Things got a lot more interesting when the economists looked at the impact of a payment of about $530 per household, an amount chosen by GiveDirectly, based on their best estimate of how much money it would take to improve people’s lives. In Rwanda, where per capita GDP is about $700, that’s a lot of money.

The study found that the larger transfer improved children’s diet, height and weight and even decreased child mortality. In an essay in The Atlantic, Michael Faye and Paul Niehaus, the co-founders of GiveDirectly, write:

Households increased their productive assets by 76 percent, saved 60 percent more, and were able to consume 32 percent more than in the past. They were able to buy more varied food for their families. Children in these households were taller, weighed more, and were less likely to die early.

As it happens, the $530 per family cost is less than what USAID spends on some other nutrition and sanitation programs in Africa, according to Zeitlin and McIntosh. So it’s not an outrageous amount to spend to prevent malnutrition or stunting.

In a summary of their findings for the Innovations for Poverty Action, a nonprofit research group, the economists write:

Particularly in places where families’ inability to afford a nutritious diet is a major factor in malnutrition, unconditional cash transfers may play a quick and effective role in improving children’s nutritional status during a critical window of development.

But….what about scale!

So long as no one is measuring impact -- and most of the time, no one is -- the incentive at agencies like USAID and among nonprofits is to “serve” as many people as possible.

Nonprofits report on how many people they trained for jobs, how many textbooks they gave out or how many chickens they gave away, none of which says anything about whether people’s lives improved.

Last spring, for example, when I visited Rwanda to report this story, a Catholic Relief Services official gave me a report saying that the Gikuriro program had served 37,887 households, resulted in the construction of 14,854 improved latrines and provided 6,388 farmers with small livestock. But if the point of all that activity was to improve child nutrition and health, it does not seem to have done much.

“Obviously scale is good, but if it is coming at the expense of impact per person, it’s something we ought to rethink,” GiveDirectly’s Paul Niehaus told me. “Setting aside the issue of cash, the question of how much we’re spending per person in any program is one the things we’re taking away from the study.”

There’s lots more to say about all this. You can read a summary of the study, a blogpost from Zeitlin and McIntosh, this story from Dylan Matthews in Vox and these thoughts from Sarah Rose and Amanda Glassman of the Center for Global Development.

A second study, already underway in Rwanda, will compare a youth workforce training program to cash; plans call for programs in Malawi, Liberia and the DRC to be evaluated rigorously as well. Kudos to USAID, GiveDirectly, Catholic Relief Services, google.org (which helped financed the research), Innovations for Poverty Action and economists Zeitlin and McIntosh for moving this important work forward.