What we don't know can hurt us

This is likely a product of confirmation bias, but I'm often reminded of how little we know about stuff that matters.  Friends with health issues visit doctors who don't know what to do. (Maybe they should do nothing. As Atul Gawande, wrote in The New Yorker in 2015: "An avalanche of unnecessary medical care is harming patients physically and financially.") We can't be sure which foods to eat and which to avoid, as the current debate over sugar reminds us. Economics, of course, is riddled with uncertainty. Will higher minimum wages eliminate jobs? No, says the U.S. Department of Labor (at least until the Trump team digs into its website). Yes, says the Congressional Budget Office, but it will lift incomes, too. Setting aside the problems caused by experts-for-hire and "fake news," scientists and social scientists operating in good faith simply cannot agree on answers to some fundamental questions.One of those questions: What should we do to help the global poor? Behind that question is another, namely, how do we know what to do to help the global poor? That is the topic of an intermittently fascinating new book called Experimental Conversations: Perspectives on Randomized Trials in Development Economics, edited by Timothy N. Ogden. It consists of a series of interviews in which development economists talk about the impact of randomized control trials (RCTs), and what they have learned from those experiments in the field--about microfinance, deworming, cash transfers, financial-literacy training and other interventions by governments and nonprofits designed to alleviate global poverty.A knowledgeable insider who is managing director of the Financial Access Initiative at NYU, Ogden collared most of the world's best-known development economists to talk about RCTs. Featured, among others, are Abhijit Banerjee, Nancy Birdsall, Chris Blattman, Alex Counts, Tyler Cowen, Angus Deaton, Esther Duflo, Rachel Glennerster,  Elie Hassenfeld, Dean Karlan, Michael Kremer, David McKenzie,  Lant Pritchett and Antoinette Schoar.The book sounds geeky and it is, but if you care about evidence and global poverty, it's well worth the effort. While the economists disagree on many things, all agree that RCTs have had an enormous impact on their field, for better or worse. RCT advocates, known as "randomistas," have reshaped thinking in academia and at such institutions as the World Bank, although it's not clear how much impact they've had on government policy.This is ironic, says Lant Prichett, an economist at Harvard's Kennedy School and an outspoken critic of RCTs, because RCTs are all about tracing cause and effect. Yet the randomistas can't explain how their work will effect policy, he says:

The claim the randomistas make is that we're going to take cash, we're going to turn it into the activity of doing RCTs, and that's going to lead to better outcomes. But they just don't have a coherent, much less evidence-based, causal model of how the outputs they produce are going to yield the outcomes they intend. That requires some positive theory of policy: how do policies actually get made, adopted and implemented? They have this just unbelievable Cro-Magnon simple model of policy adoption that essentially asserts that once there is better knowledge in the world about policies, that will lead to better policies being adopted and implemented.You could make an argument like that, but it's just wrong. So they're wrong about the fact that the outputs of RCTs are going to change policy. What's amazing about that is that precisely their critique of all other development endeavors is that they were wrong about their causal models of outputs to outcomes....It's their most powerful critique of what everyone else is doing and yet they don't have a coherent answer about what they are doing.

This is fun reading (at least for some of us!) but harsh. RCTs have made a difference. They have challenged conventional wisdom about some much-touted anti-poverty interventions, notably microcredit, while burnishing the reputation of others, including direct cash transfers and a multi-faceted effort known as the "graduation program." (See my blogpost, Trickle Up: Alleviating Extreme Poverty.)On microcredit, Jonathan Morduch, an NYU economics professor, says:

Microcredit advocates led by Muhammad Yunus had predicted the poverty rates could be slashed simply by making small loans available to microscale entrepreneurs. The claims were both enticing and specific.The randomized studies challenged those claims clearly and consistently. Studies in places as different as Morocco, Bosnia, the Philippines, Mexico and urban India showed that access to microcredit had no average impact on household incomes and only one showed an average increase--two showed a decrease--in consumption....It shot down the advocates' claims with methods that were easy to explain and difficult to impeach.

Not surprisingly, Alex Counts, founder of the Grameen Foundation, begs to differ. One value of Experimental Conversations is that it offers diverse, often-conflicting perspectives.It also functions as a reality check against the claims of foundations and NGOs that rely on anecdotes to support their programs. A couple of examples:From Abhijit Banerjee, founder of the Jameel Poverty Action Lab at MIT: "I think computer technology in education is one (intervention) where there are so many claims, so much money, so much energy, but zero effect. There is one successful RCT we did 15 years ago, and that's it. That's the extent of all successful RCTs in that space. That's a shocking fact."From MIT Sloane School prof Antoinette Schoar: "Most financial literacy training work has shown very little to no effect."By contrast, RCTs have upended people's intuitions about cash transfers and helped to fuel the growth of Give Directly, one of my favorite NGOs.As Chris Blattman, a University of Chicago economist, explains:

The evidence that's coming out from all these cash transfer studies is creating some stubborn facts. And I think these stubborn facts are having a surprising degree of influence...They're consistent across studies. They're pretty simple facts. People are not misusing money. It's cheaper (than other interventions). People seem to be happier. They use the money well. It's that all of the front-line arguments against cash are just running up against the evidence.

In response, some evidence-driven NGOs such as the International Rescue Committee are committing to more use of cash transfers.There's much more in the book, of course, and its value goes well beyond these tidbits of insights into particular interventions. It arrives as a reminder, not that we should need one, that evidence matters, that not all evidence is good evidence, and that even good evidence usually tells us less than we would want to know. (One big issue with RCTs is context, i.e., can a study in India predict outcomes in, say, Haiti?)In Timothy Ogden's conclusion, which is unavoidably inconclusive, he tells us that writing the book has left him "more skeptical about all research." In my view, that means we need more study, not less. It's also good reason for all of us to be humble about what we know and what we don't: