Adaptive Ebola vaccine trials

There's a New York Times Room for Debate feature has an excellent discussion of the ethics of trials for Ebola treatments and vaccines. Here's part of the essay by Nancy Kass and Steven Goodman:

Ethics is not just figuring out which side poses better arguments; often it’s best to find a third way. Given the breadth and deadly nature of the current Ebola outbreak, and unknowns about treatments, an "adaptive approach" seems most appropriate. Adaptive approaches allow researchers to plan a sequence of studies, or modify a single study in almost real time, as they learn more about a drug. In West Africa, for example, the first 40 Ebola patients in a trial could all get an experimental treatment, and nobody would take a placebo. If nearly all patients survived, in settings where most others were dying with the same supportive care, then it is possible that placebo testing could be avoided, and subsequent trials could randomize to different doses or treatments.

But if the results of the first trial, without placebos, revealed anything less than an almost certain cure, a design with proper controls would have to be initiated, and explained to those participating in the trial. Patients must be told that the drug is not a guaranteed life-saver, so they can see the point of the control group. (And given the multiple beliefs about Ebola among West Africans, creative approaches to promoting understanding and consent are important as well.) These placebo-controlled trials could themselves be adaptive in design, randomizing more patients to whichever therapy appears most effective, until the verdict is clear. If we are to design trials to minimize suffering and death in a whole population, we must temper our compassion with humility about what we think we know.

Ebola and health workers

It starts with familiar flu-like symptoms: a mild fever, headache, muscle and joint pains. But within days this can quickly descend into something more exotic and frightening: vomiting and diarrhoea, followed by bleeding from the gums, the nose and gastrointestinal tract.

Death comes in the form of either organ failure or low blood pressure caused by the extreme loss of fluids.

Such fear-inducing descriptions have been doing the rounds in the media lately.

However, this is not Ebola but rather Dengue Shock Syndrome, an extreme form of dengue fever, a mosquito-borne disease that struggles to make the news.

That's Seth Berkley, CEO of the GAVI Alliance, writing an opinion piece for the BBC. Berkley argues that Ebola grabs headlines not because it is particularly infectious or deadly, but because those of us from wealthy countries have otherwise forgotten what it's like to be confronted with a disease we do not know how to or cannot afford to treat.

However, in wealthy countries, thanks to the availability of modern medicines, many of these diseases can now usually be treated or cured, and thanks to vaccines they rarely have to be. Because of this blessing we have simply forgotten what it is like to live under threat of such infectious and deadly diseases, and forgotten what it means to fear them.

Ebola does combine infectiousness and rapid lethality, even with treatment, in a way that few diseases do, and it's been uniquely exoticized by books like the Hot Zone. But as Berkley and many others have pointed out, the fear isn't really justified in wealthy countries. They have health systems that can effectively contain Ebola cases if they arrive -- which I'd guess is more likely than not. So please ignore the sensationalism on CNN and elsewhere. (See for example Tara Smith on other cases when hemorraghic fevers were imported into the US and contained.)

But one way that Ebola is different -- in degree if not in kind -- to the other diseases Berkley cites (dengue, measles, childhood diseases) is that its outbreaks are both symptomatic of weak health systems and then extremely destructive to the fragile health systems that were least able to cope with it in the first place.

Like the proverbial canary in the coal mine, an Ebola outbreak reveals underlying weaknesses in health systems. Shelby Grossman highlights this article from Africa Confidential:

MSF set up an emergency clinic in Kailahun [Sierra Leone] in June but several nurses had already died in Kenema. By early July, over a dozen health workers, nurses and drivers in Kenema had contracted Ebola and five nurses had died. They had not been properly equipped with biohazard gear of whole-body suit, a hood with an opening for the eyes, safety goggles, a breathing mask over the mouth and nose, nitrile gloves and rubber boots.

On 21 July, the remaining nurses went on strike. They had been working twelve-hour days, in biohazard suits at high temperatures in a hospital mostly without air conditioning. The government had promised them an extra US$30 a week in danger money but despite complaints, no payment was made. Worse yet, on 17 June, the inexperienced Health and Sanitation Minister, Miatta Kargbo, told Parliament that some of the nurses who had died in Kenema had contracted Ebola through promiscuous sexual activity.

Only one nurse showed up for work on 22 July, we hear, with more than 30 Ebola patients in the hospital. Visitors to the ward reported finding a mess of vomit, splattered blood and urine. Two days later, Khan, who was leading the Ebola fight at the hospital and now with very few nurses, tested positive. The 43-year-old was credited with treating more than 100 patients. He died in Kailahun at the MSF clinic on 29 July...

In addition to the tragic loss of life, there's also the matter of distrust of health facilities that will last long after the epidemic is contained. Here's Adam Nossiter, writing for the NYT on the state of that same hospital in Kenema as of two days ago:

The surviving hospital workers feel the stigma of the hospital acutely.

“Unfortunately, people are not coming, because they are afraid,” said Halimatu Vangahun, the head matron at the hospital and a survivor of the deadly wave that decimated her nursing staff. She knew, all throughout the preceding months, that one of her nurses had died whenever a crowd gathered around her office in the mornings.

There's much to read on the current outbreak -- see also this article by Denise Grady and Sheri Fink (one of my favorite authors) on tracing the index patient (first case) back to a child who died in December 2013. One of the saddest things I've read about previous Ebola outbreaks is this profile of Dr. Matthew Lukwiya, a physician who died fighting Ebola in Uganda.

The current outbreak is different in terms of scale and its having reached urban areas, but if you read through these brief descriptions of past Ebola outbreaks (via Wikipedia) you'll quickly see that the transmission to health workers at hospitals is far too typical. Early transmission seems to be amplified by health facilities that weren't properly equipped to handle the disease. (See also this article article (PDF) on a 1976 outbreak.) The community and the brave health workers responding to the epidemic then pay the price.

Ebola's toll on health workers is particularly harsh given that the affected countries are starting with an incredible deficit. I was recently looking up WHO statistics on health worker density, and it struck me that the three countries at the center of the current Ebola outbreak are all close to the very bottom of rankings by health worker density. Here's the most recent figures for the ratio of physicians and nurses to the population of each country:* 

Liberia has already lost three physicians to Ebola, which is especially tragic given that there are so few Liberian physicians to begin with: somewhere around 60 (in 2008). The equivalent health systems impact in the United States would be something like losing 40,000 physicians in a single outbreak.

After the initial emergency response subsides -- which will now be on an unprecedented scale and for an unprecedented length of time -- I hope donors will make the massive investments in health worker training and systems strengthening that these countries needed prior to the epidemic. More and better trained and equipped health workers will save lives otherwise lost to all the other infectious diseases Berkley mentioned in the article linked above, but they will also stave off future outbreaks of Ebola or new diseases yet unknown. And greater investments in health systems years ago would have been a much less costly way -- in terms of money and lives -- to limit the damage of the current outbreak.  

(*Note on data: this is quick-and-dirty, just to illustrate the scale of the problem. Ie, ideally you'd use more recent data, compare health worker numbers with population numbers from the same year, and note data quality issues surrounding counts of health workers)

(Disclaimer: I've remotely supported some of CHAI's work on health systems in Liberia, but these are my personal views.)

Have recent global health gains gone to the poor?

Have recent global gains gone to the poor in developing countries? Or the relatively rich? An answer:

We find that with the exception of HIV prevalence, where progress has, on average, been markedly pro-rich, progress on the MDG health outcome (health status) indicators has, on average, been neither pro-rich nor pro-poor. Average rates of progress are similar among the poorest 40 percent and among the richest 60 percent.

That's Adam Wagstaff, Caryn Bredenkamp, and Leander Buisman in a new article titled "Progress on Global Health Goals: are the Poor Being Left Behind?" (full text here). The answer seems to be "mostly no, sometimes yes", but the exceptions to the trend are as important as the trend itself.

I originally flagged this article to read because Wagstaff is one of the authors, and I drew on a lot of his work for my masters thesis (which looked at trends in global health inequities in Ethiopia). One example is this handy World Bank report (PDF) which is a how-to for creating concentration indexes and other measures of inequality, complete with Stata. A concentration index is essentially a health inequality version of the Gini index: instead of showing the concentration of wealth by wealth, or income by income, you measure the concentration of some measure of health by a measure of wealth or income, often the DHS wealth index since it's widely available.

If your chosen measure of health -- let's say, infant mortality -- doesn't vary by wealth, then you'd graph a straight line at a 45 degree angle -- sometimes called the line of equality. But in most societies the poor get relatively more of a bad health outcome (like mortality) and rather less of good things like access to vaccination. In both cases the graphed line would be a curve that differs from the line of equality, which is called a concentration curve. The further away from the line of equality the concentration curve is, the more unequal the distribution of the health outcome is. And the concentration index is simply twice the area between the two lines (again, the Gini index is the equivalent number when comparing income vs. income). The relationship between the two is illustrated in this example graph from my thesis:

You can also just compare, say, mortality rates for the top and bottom quintiles of the wealth distribution, or comparing the top 1% vs. bottom 99%, or virtually any other division, but all of those measures essential ignore a large amount of information in middle of the distribution, or require arbitrary cutoffs. The beauty of concentration curves and indexes is that they use all available information. An even better approach is to use multiple measures of inequality and see if the changes you see are sensitive to your choice of measures; it's a more a convincing case if they're not.

The new Wagstaff, Bredenkamp, and Buisman paper uses such concentration indexes, and other measures of inequity, to "examine differential progress on health Millennium Development Goals (MDGs) between the poor and the better off within countries." They use a whopping 235 DHS and MICs surveys between 1990-2011, and find the following:

On average, the concentration index (the measure of relative inequality that we use) neither rose nor fell. A rosier picture emerges for MDG intervention indicators: whether we compare rates of change for the poorest 40 percent and richest 60 percent or consider changes in the concentration index, we find that progress has, on average, been pro-poor.

However, behind these broad-brush findings lie variations around the mean. Not all countries have progressed in an equally pro-poor way. In almost half of countries, (relative) inequality in child malnutrition and child mortality fell, but it also increased in almost half of countries, often quite markedly.We find some geographic concentration of pro-rich progress; in almost all countries in Asia, progress on underweight has been pro-rich, and in much of Africa, inequalities in under-five mortality have been growing. Even on the MDG intervention indicators, we find that a sizable fraction of countries have progressed in a pro-rich fashion.

They also compared variations that were common across countries vs. common across indicators -- in other words, to see whether the differences across countries and indicators were because, say, some health interventions are just easier to reach the poorest with, and found that more of the variation came from differences between countries, rather than differences between indicators.

One discussion point they stress is that it's been easier to promote equality in interventions, rather than equality in outcomes, and that part of the story is related to the quality of care that poorer citizens receive. From the discussion:

One hypothesis is that the quality of health care is worse for lower socioeconomic groups; though the poorest 40 percent may have experienced a larger percentage increase in, for example, antenatal visits, they have not observed the same improvement in the survival prospects of their babies. If true, this finding would point to the need for a monitoring framework that captures not only the quantity of care (as is currently the case) but also its quality.

Is there a global health bubble? (Or: should you get an MPH?)

There's a LinkedIn group for Global Public Health that occasionally has good discussion. One example, albeit a sobering one, is the current discussion of employment opportunities after MPH. I've been meaning to write about jobs for a while because now that I'm on the other side of the picture -- an employed professional with a job at a reputable organization, rather than a grad student -- I find myself doing an increasing number of informational interviews, and saying much the same thing each time.

[First, some caveats on the generalizability of the advice below: first, folks with an MPH from another country often have less debt burden than Americans, so may find it easier to do long unpaid or underpaid internships. Second, folks from low- to middle-income countries are and should be more employable, especially in their own countries. Why? Because they have incredibly valuable linguistic and cultural talents (see Alanna Shaikh's recent post on this), so much so that an organization choosing between an outsider and a local with the same technical skills, communication skills, etc, should almost always choose the local. If they don't, that's generally a sign of a dysfunctional or discriminatory organizations.]

The problem is that there is something of an MPH bubble, especially in global health. The size of MPH classes has increased and - more importantly - the number of schools granting degrees has risen rapidly. Degrees focusing on global health also seem to be growing faster than the rest of the field.  (I'd welcome data on class size and jobs in the industry if anyone knows where to find it.) This is happening in part because public health attracts a lot of idealists who are interested in the field because they want to make a difference, rather than rationally choosing between the best paying jobs, and global health has gotten a lot of good press over the last decade. Call this the Mountains Beyond Mountains Effect if you like.

If you know this, and still go into the field, and don't have an MD or PhD that qualifies you for a different sort of job altogether, then you need to distinguish yourself from the crowd to be employable. I'm assuming your goal is to get a good job in global health, where "good job" is defined as a full-time professional position with a good (not necessarily big-name) organization, working on fulfilling projects and being paid well enough to live comfortably while paying off the loans that most American MPH grads will have. For some, though not all, a good job might also mean one that's either based abroad or involves frequent international travel. If that's the goal, then there are several ways to distinguish yourself:

1. get some sort of hard, transferable skills. This can be research or M&E skills, especially quantitative data crunching ability, or it can be management/coordination experience with serious responsibility. Or other things. The key point is that your skillset should match jobs that are out there, and be something that not everyone has. A lot of MPH programs feature concentrations -- or the lack thereof -- that are more appealing to students than they are to employers. A biostatistics concentration will likely serve you better than a global health concentration, for instance, and with some exceptions.
2. get solid international experience, preferably a year or more. Professional experience in public health -- even with a lesser-known organization -- is much more valuable than experience teaching, or studying abroad. Travel doesn't count much, and it's better to have experience in the region you're interested in working in. There's a huge catch-22 here, as you need international experience to get it, so that many global health folks start off doing work they're critical of later in their careers.
3. relatedly, speak an in-demand language, though this will only help you to work in the region where it's spoken.
4. have professional work experience. Even if it's not in global health, having worked an office job for a year or two makes you more desirable to employers. No one wants to be your first employer, so folks who go straight to an MPH may find themselves less employable than peers who worked for a bit first.
5. go to a top school, which signals that you're smarter or better qualified than others (this often isn't true, the key part is the signalling, and the networks you acquire). Also, graduates of top schools often get good jobs in part because those schools select people with good work experience, skills, and connections to begin with, so that a superior candidate at a school that's perceived to be a second or third-tier school can do just fine.
6. avoid debt (which often conflicts with 'go to a top school') to give yourself the flexibility to work for less or for nothing at first, until you can do the above.

Any one or two items from this list probably won't cut it: you need to acquire several.  For example, I've known peers with a solid technical degree from a top school and some international experience who still struggled to get jobs at first because they had never had a regular office job before grad school. Also, the relative importance of each will vary according to the subfield of global health you're interested in. For instance, learning languages might be more important for an implementation person (program coordinator or manager) or a qualitative researcher than it is for a data cruncher.

I used to be pre-med, until I realized I was more interested in policy and did not want to be a clinician, and the path to doing so in the US is long and expensive. Like many former pre-med students who decided not to go to medical school, it took me a while to figure out what I wanted to do, and how to do that without an MD. A couple years post-undergrad I found myself working a job that was interesting enough but not what I ultimately wanted to do, and unable to get a first job in global health without the requisite skills or longer international experience, and I didn't have the resources to just up and move abroad on my own. So, I went to go to grad school with a technical focus (epidemiology) at a top school, and then used the practicum requirement to build more international experience (Ethiopia). The combination of school and work experience gave me solid quantitative skills because I chose to focus on that each step of the way. But, it also meant taking on quite a bit of debt, and the international practicum would have required even more had I not had generous funding from the econ/policy degree I did. This has worked out well for me, though that same path won't necessarily work for everyone -- especially if you have different interests from mine! -- and I think it's instructive enough to share.

The upside of this bubble is that organizations often hire well-educated, experienced people for even entry level position. The downside is that people from less privileged educational or financial backgrounds often get blocked out of the sector, given that you might have to volunteer for an extended period of time to get the requisite experience, or take on a lot of debt to get a good graduate degree.

In conclusion, getting an MPH -- and trying to break into global health -- is a personal decision that might work out differently depending on your personal goals, the lifestyle you're looking for, and your financial background. But if you do get one, be aware that the job market is not the easiest to navigate, and many MPH grads end up unemployed or underemployed for a stretch. Focus on acquiring the skills and experience that will make organizations want to hire you.

Data: big, small, and meta

When I read this New York Times piece back in August, I was in the midst of preparation and training for data collection at rural health facilities in Zambia. The Times piece profiles a group called Global Pulse that is doing good work on the 'big data' side of global health:

The efforts by Global Pulse and a growing collection of scientists at universities, companies and nonprofit groups have been given the label “Big Data for development.” It is a field of great opportunity and challenge. The goal, the scientists involved agree, is to bring real-time monitoring and prediction to development and aid programs. Projects and policies, they say, can move faster, adapt to changing circumstances and be more effective, helping to lift more communities out of poverty and even save lives.

Since I was gearing up for 'field work' (more on that here; I'll get to it soon), I was struck at the time by the very different challenges one faces at the other end of the spectrum. Call it small data? And I connected the Global Pulse profile with this, by Wayan Vota, from just a few days before:

The Sneakernet Reality of Big Data in Africa

When I hear people talking about “big data” in the developing world, I always picture the school administrator I met in Tanzania and the reality of sneakernet data transmissions processes.

The school level administrator has more data than he knows what to do with. Years and years of student grades recorded in notebooks – the hand-written on paper kind of notebooks. Each teacher records her student attendance and grades in one notebook, which the principal then records in his notebook. At the local district level, each principal’s notebook is recorded into a master dataset for that area, which is then aggregated at the regional, state, and national level in even more hand-written journals... Finally, it reaches the Minister of Education as a printed-out computer-generated report, complied by ministerial staff from those journals that finally make it to the ministry, and are not destroyed by water, rot, insects, or just plain misplacement or loss. Note that no where along the way is this data digitized and even at the ministerial level, the data isn’t necessarily deeply analyzed or shared widely....

And to be realistic, until countries invest in this basic, unsexy, and often ignored level of infrastructure, we’ll never have “big data” nor Open Data in Tanzania or anywhere else. (Read the rest here.)

Right on. And sure enough two weeks later I found myself elbow-deep in data that looked like this -- "Sneakernet" in action:

In many countries a quite a lot of data -- of varying quality -- exists, but it's often formatted like the above. Optimistically, it may get used for local decisions, and eventually for high-level policy decisions when it's months or years out of date. There's a lot of hard, good work being done to improve these systems (more often by residents of low-income countries, sometimes by foreigners), but still far too little. This data is certainly primary, in the sense that was collected on individuals, or by facilities, or about communities, but there are huge problems with quality, and with the sneakernet by which it gets back to policymakers, researchers, and (sometimes) citizens.

For the sake of quick reference, I keep a folder on my computer that has -- for each of the countries I work in -- most of the major recent ultimate sources of nationally-representative health data. All too often the only high-quality ultimate source is the most recent Demographic and Health Survey, surely one of the greatest public goods provided by the US government's aid agency. (I think I'm paraphrasing Angus Deaton here, but can't recall the source.) When I spent a summer doing epidemiology research with the New York City Department of Health and Mental Hygiene, I was struck by just how many rich data sources there were to draw on, at least compared to low-income countries. Very often there just isn't much primary data on which to build.

On the other end of the spectrum is what you might call the metadata of global health. When I think about the work the folks I know in global health -- classmates, professors, acquaintances, and occasionally thought not often me -- do day to day, much of it is generating metadata. This is research or analysis derived from the primary data, and thus relying on its quality. It's usually smart, almost always well-intentioned, and often well-packaged, but this towering edifice of effort is erected over a foundation of primary data; the metadata sometimes gives the appearance of being primary, when you dig down the sources often point back to those one or three ultimate data sources.

That's not to say that generating this metadata is bad: for instance, modeling impacts of policy decisions given the best available data is still the best way to sift through competing health policy priorities if you want to have the greatest impact. Or a more cynical take: the technocratic nature of global health decision-making requires that we either have this data or, in its absence, impute it. But regardless of the value of certain targeted bits of the metadata, there's the question of the overall balance of investment in primary vs. secondary-to-meta data, and my view -- somewhat ironically derived entirely from anecdotes -- is that we should be investing a lot more in the former.

One way to frame this trade-off is to ask, when considering a research project or academic institute or whatnot, whether the money spent on that project might result in more value for money if it was spent instead training data collectors and statistics offices, or supporting primary data collection (e.g., funding household surveys) in low-income countries. I think in many cases the answer will be clear, perhaps to everyone except those directly generating the metadata.

That does not mean that none of this metadata is worthwhile. On the contrary, some of it is absolutely essential. But a lot isn't, and there are opportunity costs to any investment, a choice between investing in data collection and statistics systems in low-income countries, vs. research projects where most of the money will ultimately stay in high-income countries, and the causal pathway to impact is much less direct.  

Looping back to the original link, one way to think of the 'big data' efforts like Global Pulse is that they're not metadata at all, but an attempt to find new sources of primary data. Because there are so few good sources of data that get funded, or that filter through the sneakernet, the hope is that mobile phone usage and search terms and whatnot can be mined to give us entirely new primary data, on which to build new pyramids of metadata, and with which to make policy decisions, skipping the sneakernet altogether. That would be pretty cool if it works out.

A more useful aid debate

Ken Opalo highlights recent entries on the great aid debate from Bill Gates, Jeff Sachs, Bill Easterly, and Chris Blattman. Much has been said on this debate, and sometimes it feels like it's hard to add anything new. But since having a monosyllabic first name seem sufficient qualification to weigh in, I will. First, this part of Ken's post resonates with me:

I think most reasonable people would agree that Sachs kind of oversold his big push idea in The End of Poverty. Or may be this was just a result of his attempt to shock the donor world into reaching the 0.7 percent mark in contributions. In any event it is unfortunate that the debate on the relative efficacy of aid left the pages of journal articles in its current form. It would have been more helpful if the debate spilled into the public in a policy-relevant form, with questions like: under what conditions does aid make a difference? What can we do to increase the efficacy of aid? What kinds of aid should we continue and what kinds should we abolish all together? (emphasis added)

Lee Crawfurd wrote something along these lines too: "Does Policy Work?"  Lee wrote that on Jan 10, 2013, and I jokingly said it was the best aid blog post of the year (so far). Now that 2013 has wrapped up, I'll extend that evaluation to 'best aid blog post of 2013'. It's worth sharing again:

The question "does policy work" is jarring, because we immediately realise that it makes little sense. Governments have about 20-30 different Ministries, which immediately implies at least 20-30 different areas of policy. Does which one work? We have health and education policy, infrastructure policy (roads, water, energy), trade policy, monetary policy, public financial management, employment policy, disaster response, financial sector policy, climate and environment policy, to name just a few. It makes very little sense to ask if they all collectively "work" or are "effective". Foreign aid is similar. Aid supports all of these different areas of policy....

A common concern is about the impact of aid on growth... Some aid is specifically targeted at growth - such as financing infrastructure or private sector development. But much of it is not. One of the few papers which looks at the macroeconomic impact of aid and actually bothers to disaggregate even a little the different types of aid, finds that the aid that could be considered to have growth as a target, does increase growth. It's the aid that was never intended to impact growth at all, such as humanitarian assistance, which doesn't have any impact on growth.

I like to think that most smart folks working on these issues -- and that includes both Sachs and Easterly -- would agree with the following summaries of our collective state of knowledge:

•  A lot of aid projects don't work, and some of them do harm.
• Some aid, especially certain types of health projects, works extremely well.

The disagreement is on the balance of good and bad, so I wish -- as Ken wrote -- the debate spilled into the public sphere along those lines (which is good? which is bad? how can we get a better mix?) rather than the blanket statements both sides are driven to by the very publicness of the debate. It reminds me a bit of debates in theology: if you put a fundamentalist and Einstein in the same room, they'll both be talking about "God" but meaning very different things with the same words. (This is not a direct analogy, so don't ask who is who...)

When Sachs and Easterly talk about whether aid "works", it would be nice if we could get everyone to first agree on a definition of "aid" and "works". But much of this seems to be driven by personal animosity between Easterly and Sachs, or more broadly, by personal animosity of a lot of aid experts vs. Sachs. Why's that? I think part of the answer is that it's hard to tell when Sachs is trying to be a scientist, and when he's trying to be an advocate. He benefits from being perceived as the former, but in reality is much more the latter. Nina Munk's The Idealist -- an excellent profile of Sachs I've been meaning to review -- explores this tension at some length. The more scientifically-minded get riled up by this confusion -- rightfully, I think. At the same time, public health folks tend to love Sachs precisely because he's been a powerful advocate for some types of health aid that demonstrably work -- also rightfully, I think. There's a tension there, and it's hard to completely dismiss one side as wrong, because the world is complicated and there are many overlapping debates and conversations; academic and lay, public and private, science and advocacy.

So, back to Ken's questions that would be answered by a more useful aid debate:

• Under what conditions does aid make a difference?
• What can we do to increase the efficacy of aid?
• What kinds of aid should we continue and what kinds should we abolish all together?

Wouldn't it be amazing if the public debate were focused on these questions? Actually, something like that was done: Boston Review had a forum a while back on "Making Aid Work" with responses by Abhijit Banerjee, Angus Deaton, Howard White, Ruth Levine, and others. I think that series of questions is much more informative than another un-moderated round of Sachs vs Easterly.

Spreading the word

If you haven't already read Atul Gawande's latest New Yorker piece on why some ideas spread fast and other spread slow, get to it:

 In the era of the iPhone, Facebook, and Twitter, we’ve become enamored of ideas that spread as effortlessly as ether. We want frictionless, “turnkey” solutions to the major difficulties of the world—hunger, disease, poverty. We prefer instructional videos to teachers, drones to troops, incentives to institutions. People and institutions can feel messy and anachronistic. They introduce, as the engineers put it, uncontrolled variability.

But technology and incentive programs are not enough. “Diffusion is essentially a social process through which people talking to people spread an innovation,” wrote Everett Rogers, the great scholar of how new ideas are communicated and spread. Mass media can introduce a new idea to people. But, Rogers showed, people follow the lead of other people they know and trust when they decide whether to take it up. Every change requires effort, and the decision to make that effort is a social process.

Much of the material is Gawande's essay won't be new if you're already interested in or working on maternal and child health, but Gawande presents it incredibly well. His comparison of spreading social innovation with the work of salesman also reminded me of another parallel: the parallels between diffusing secular, health-enhancing ideas and missionaries' evangelistic techniques.

If that last sentence scares you off, hold on a moment for some background. I grew up in a small religious town in Arkansas and my first trips to developing countries were as a missionary. Over time my interests shifted from the preaching and teaching side of things to the medical side, and eventually to health and development policy as an entirely secular pursuit. When I first got to grad school for public health this resulted in some awkward moments, as many conversations would start with "so what first interested you in global health?" If I led with "well, I grew up wanting to be a missionary" I would often get one of two reactions: immediate skepticism of my motivations from my secular liberal classmates, or enthusiastic endorsement of my work (as they misunderstood it) from religious classmates. All that to say: while I think there are very good general reasons to keep public health and missionary efforts as separate as possible, both in theory and praxis, there are several things we secular liberals can still learn from the more devout.

One example is the neverending debates amongst evangelists between those who seek technological shortcuts and those who stick with old-fashioned person-to-person contact. This is a frequent topic at missions conferences (if you didn't know such conferences existed, it might be an interesting google). You can view the rise of Christian radio broadcasts, followed by Christian TV and televangelists, as the great technological shortcuts: they give a single preacher the ability to reach millions, and if the message is just as good as when delivered in person, why shouldn't it be just as effective? Some people are persuaded by televangelists, of course, but the effectiveness of the individual doesn't scale easily to mass media. Likewise, in recent years there's been much enthusiasm for social media and its potential to save more souls -- but the results rarely pan out.  So despite all of the advances in mass and social media, most evangelists still harp on the importance of individual contact, of building relationships. One of the most effective (in terms of growth rate) groups in the world are Mormons, who, no coincidence, devote years of effort to one-on-one contact.

Gawande's essay tells the story of how BRAC precipitated oral rehydration solution in Bangladesh, and I couldn't help thinking of their campaign  as a sort of especially successful roving gospel meeting. And here's Gawande's closing, where he talks with a nurse who was convinced by a younger, less-experienced trainer to adopt some best practices for safe childbirth:

 “Why did you listen to her?” I [Gawande] asked. “She had only a fraction of your experience.”

In the beginning, she didn’t, the nurse admitted. “The first day she came, I felt the workload on my head was increasing.” From the second time, however, the nurse began feeling better about the visits. She even began looking forward to them.

“Why?” I asked.

All the nurse could think to say was “She was nice.”

“She was nice?”

“She smiled a lot.”

“That was it?”

“It wasn’t like talking to someone who was trying to find mistakes,” she said. “It was like talking to a friend.”

Shortcuts are nice: in public health, unlike evangelism, it's usually actions rather than beliefs that ultimately count, so I'm all for technological shortcuts when they're available and effective. But they're too few and far between, and much of the low-hanging fruit in global health has already been picked. To climb the next step require a lot more effort at improving the "messy and anachronistic"  processes of people and institutions.

Formalizing corruption: US medical system edition

Oh, corruption. It interferes with so many aspects of daily life, adding time to the simplest daily tasks, costing more money, and -- often the most frustrating aspect -- adding huge doses of uncertainty. That describes life in many low-income, high-corruption countries, leading to many a conversation with friends about comparisons with the United States and other wealthy countries. How did the US "solve" corruption? I've heard (and personally made) the argument that the US reduced corruption at least in part by formalizing it; by channeling the root of corruption, a sort of rent-seeking on a personal level, to rent-seeking on an institutional level. The US political and economic system has evolved such that some share of any wealth created is channeled into the pockets of a political and economic elite who benefit from the system and in turn reinforce it. That unproductively-channeled share of wealth is simultaneously a) probably smaller than the share of wealth lost to corruption in most developing countries, b) still large enough to head off -- along with the threat of more effective prosecution -- at least some more overt corruption, and c) still a major drain on society.

An example: Elisabeth Rosenthal profiles medical tourism in an impressive series in the New York Times. In part three of the series, an American named Michael Shopenn travels to Belgium to get a hip replacement. Why would he need to? Because health economics in the US is less a story of free markets and  more a story of political capture by medical interests, including technology and pharmaceutical companies, physicians' groups, and hospitals:

Generic or foreign-made joint implants have been kept out of the United States by trade policy, patents and an expensive Food and Drug Administration approval process that deters start-ups from entering the market. The “companies defend this turf ferociously,” said Dr. Peter M. Cram, a physician at the University of Iowa medical school who studies the costs of health care.

Though the five companies make similar models, each cultivates intense brand loyalty through financial ties to surgeons and the use of a different tool kit and operating system for the installation of its products; orthopedists typically stay with the system they learned on. The thousands of hospitals and clinics that purchase implants try to bargain for deep discounts from manufacturers, but they have limited leverage since each buys a relatively small quantity from any one company.

In addition, device makers typically require doctors’ groups and hospitals to sign nondisclosure agreements about prices, which means institutions do not know what their competitors are paying. This secrecy erodes bargaining power and has allowed a small industry of profit-taking middlemen to flourish: joint implant purchasing consultants, implant billing companies, joint brokers. There are as many as 13 layers of vendors between the physician and the patient for a hip replacement, according to Kate Willhite, a former executive director of the Manitowoc Surgery Center in Wisconsin.

If this system existed in another country we wouldn't hesitate to call it corrupt, and to note that it actively hurts consumers. It should be broken up by legislation for the public good, but instead it's protected by legislators who are lobbied by the industry and by doctors who receive kickbacks, implicit and explicit. Contrast that with the Belgian system:

His joint implant and surgery in Belgium were priced according to a different logic. Like many other countries, Belgium oversees major medical purchases, approving dozens of different types of implants from a selection of manufacturers, and determining the allowed wholesale price for each of them, for example. That price, which is published, currently averages about $3,000, depending on the model, and can be marked up by about $180 per implant. (The Belgian hospital paid about $4,000 for Mr. Shopenn’s high-end Zimmer implant at a time when American hospitals were paying an average of over $8,000 for the same model.)

“The manufacturers do not have the right to sell an implant at a higher rate,” said Philip Boussauw, director of human resources and administration at St. Rembert’s, the hospital where Mr. Shopenn had his surgery. Nonetheless, he said, there was “a lot of competition” among American joint manufacturers to work with Belgian hospitals. “I’m sure they are making money,” he added.

It's become a cliche to compare the US medical system to European ones, but those comparisons are made because it's hard to realize just how systematically corrupt -- and expensive, as a result -- the US system is without comparing it to ones that do a better job of channeling the natural profit-seeking goals of individuals and companies towards the public good. (For the history of how we got here, Paul Starr is a good place to start.)

The usual counterargument for protecting such large profit margins in the US is that they drive innovation, which is true but only to an extent. And for the implants industry that argument is much less compelling since many of the newer, "innovative" products have proved somewhere between no better and much worse in objective tests.

The Times piece is definitely worth a read. While I generally prefer the formalized corruption to the unformalized version, I'll probably share this article with friends -- in Nigeria, or Ethiopia, or wherever else the subject comes up next.

Advocates and scientists

A new book by The Idealist: Jeffrey Sachs and the Quest to End Poverty. The blurbs on Amazon are fascinating because they indicate that either the reviewers didn't actually read the book (which wouldn't be all that surprising) or that Munk's book paints a nuanced enough picture that readers can come away with very different views on what it actually proves. Here are two examples:

Amartya Sen: “Nina Munk’s book is an excellent – and moving – tribute to the vision and commitment of Jeffrey Sachs, as well as an enlightening account of how much can be achieved by reasoned determination.”

Robert Calderisi: "A powerful exposé of hubris run amok, drawing on touching accounts of real-life heroes fighting poverty on the front line."

The publisher's description seems to encompass both of those points of view: "The Idealist is the profound and moving story of what happens when the abstract theories of a brilliant, driven man meet the reality of human life." That sounds like a good read to me -- I look forward to reading when it comes out in September.

Munk's previous reporting strikes a similar tone. For example, here's an excerpt of her 2007 Vanity Fair profile of Sachs:

Leaving the region of Dertu, sitting in the back of an ancient Land Rover, I'm reminded of a meeting I had with Simon Bland, head of Britain's Department for International Development in Kenya. Referring to the Millennium Villages Project, and to Sachs in particular, Bland laid it out for me in plain terms: "I want to say, 'What concept are you trying to prove?' Because I know that if you spend enough money on each person in a village you will change their lives. If you put in enough resources—enough foreigners, technical assistance, and money—lives change. We know that. I've been doing it for years. I've lived and worked on and managed [development] projects.

"The problem is," he added, "when you walk away, what happens?"

Someone -- I think it was Chris Blattman, but I can't find the specific post -- wondered a while back whether too much attention has been given to the Millennium Villages Project. After all, the line of thinking goes, the MVP's have really just gotten more press and aren't that different from the many other projects with even less rigorous evaluation designs. That's certainly true: when journalists and aid bloggers debate the MVPs, part of what they're debating is Sachs himself because he's such a polarizing personality. If you really care about aid policy, and the uses of evidence in that policy, then that can all feel like an unhelpful distraction. Most aid efforts don't get book-length profiles, and the interest in Sachs' personality and persona will probably drive the interest in Munk's book.

But I also think the MVP debates have been healthy and interesting -- and ultimately deserving of most of the heat generated -- because they're about a central tension within aid and development, as well as other fields where research intersects with activism. If you think we already generally know what to do, then it makes sense to push forward with it at all costs. The naysayers who doubt you are unhelpful skeptics who are on some level ethically culpable for blocking good work. If you think the evidence is not yet in, then it makes more sense to function more like a scientist, collecting the evidence needed to make good decisions in the longer term. The naysayers opposing the scientists are then utopian advocates who throw millions at unproven projects. I've seen a similar tension within the field of public health, between those who see themselves primarily as advocates and those who see themselves as scientists, and I'm sure it exists elsewhere as well.

That is, of course, a caricature -- few people fall completely on one side of the advocates vs. scientists divide. But I think the caricature is a useful one for framing arguments. The fundamental disagreement is usually not about whether evidence should be used to inform efforts to end poverty or improve health or advance any other goal. Instead, the disagreement is often over what the current state of knowledge is. And on that note, if you harbor any doubts on where Sachs has positioned himself on that spectrum here's the beginning of Munk's 2007 profile:

In the respected opinion of Jeffrey David Sachs.... the problem of extreme poverty can be solved. In fact, the problem can be solved "easily." "We have enough on the planet to make sure, easily, that people aren't dying of their poverty. That's the basic truth," he tells me firmly, without a doubt.

...To Sachs, the end of poverty justifies the means. By hook or by crook, relentlessly, he has done more than anyone else to move the issue of global poverty into the mainstream—to force the developed world to consider his utopian thesis: with enough focus, enough determination, and, especially, enough money, extreme poverty can finally be eradicated.

Once, when I asked what kept him going at this frenzied pace, he snapped back, "If you haven't noticed, people are dying. It's an emergency."

----

via Gabriel Demombynes.

If you're new to the Millennium Villages debate, here's some background reading: a recent piece in Foreign Policy by Paul Starobin, and some good posts by Chris Blattman (one, two, three), this gem from Owen Barder, and Michael Clemens.

Uninformative paper titles: "in Africa"

When I saw a new NBER working paper titled "Disease control, demographic change and institutional development in Africa" (PDF) pop up in the NBER RSS feed I thought the title sounded interesting, so I downloaded the paper to peruse later. Then today the new-ish (and great!) blog Cherokee Gothic highlighted the same paper in a post, and I finally took a look. Unfortunately the paper title is rather uninformative, as the authors only used data from Burkina Faso. Sure, economics papers tend to have bigger, less formal titles than papers in some other fields, but I think this is particularly unhelpful. There are enough search frictions in finding applicable literature on any given topic that it helps to be somewhat more precise.

For reference, here's Burkina Faso:

And here's Africa:

Not the same.

It's unclear from the data and arguments presented how these results -- for a regional disease control program, but only using data from Burkina Faso -- might generalize to the quite diverse disease environments, demographic trends, and institutional histories of various African countries. The paper doesn't answer or even give much grounds for speculation on whether onchocerciasis or other disease control programs would yield similar results in countries as diverse as (for example) Senegal, Ethiopia, Uganda, and Angola.

A quick thought experiment: Virginia's population is about 1.5% of the total population of North America, just as Burkina Faso's population is about 1.5% of the total population on Africa. Can you imagine someone writing a paper on health and institutions using data from Virginia and titling that paper "Health and institutions in North America"? Or writing a paper on Vietnamese history and titling it "A history of Asia"? Probably not.

"Redefining global health delivery"

Jim Yong Kim, Paul Farmer, and Michael Porter wrote a piece called "Redefining global health delivery" for the Lancet in May. The abstract:

Initiatives to address the unmet needs of those facing both poverty and serious illness have expanded significantly over the past decade. But many of them are designed in an ad-hoc manner to address one health problem among many; they are too rarely assessed; best practices spread slowly. When assessments of delivery do occur, they are often narrow studies of the cost-effectiveness of a single intervention rather than the complex set of them required todeliver value to patients and their families. We propose a framework for global health-care delivery and evaluation by considering efforts to introduce HIV/AIDS care to resource-poor settings. The framework introduces the notion of care delivery value chains that apply a systems-level analysis to the complex processes and interventions that must occur, across a health-care system and over time, to deliver high-value care for patients with HIV/AIDS and cooccurring conditions, from tuberculosis to malnutrition. To deliver value, vertical or stand-alone projects must be integrated into shared delivery infrastructure so that personnel and facilities are used wisely and economies of scale reaped. Two other integrative processes are necessary for delivering and assessing value in global health: one is the alignment of delivery with local context by incorporating knowledge of both barriers to good outcomes (from poor nutrition to a lack of water and sanitation) and broader social and economic determinants of health and wellbeing (jobs, housing, physical infrastructure). The second is the use of effective investments in care delivery to promote equitable economic development, especially for those struggling against poverty and high burdens of disease. We close by reporting our own shared experience of seeking to move towards a science of delivery by harnessing research and training to understand and improve care delivery.

I think the overall thrust of the piece is something that is widely agreed upon by global health policy wonks, but I like that they lay out a more specific framework for thinking with this sort of systems approach. But, I'd love to see some more detail on putting it into practice on a national or subnational level.

Slow down there

Max Fisher has a piece in the Washington Post presenting "The amazing, surprising, Africa-driven demographic future of the Earth, in 9 charts". While he notes that the numbers are "just projections and could change significantly under unforeseen circumstances" the graphs don't give any sense of the huge uncertainty involved in projecting trends out 90 years in the future. Here's the first graph:

The population growth in Africa here is a result of much higher fertility rates, and a projected slower decline in those rates.

But those projected rates have huge margins of error. Here's the total fertility rate, or "the average number of children that would be born to a woman over her lifetime"  for Nigeria, with confidence intervals that give you a sense of just how little we know about the future:

That's a lot of uncertainty! (Image from here, which I found thanks to a commenter on the WaPo piece.)

It's also worth noting that if you had made similar projections 87 years ago, in 1926, it would have been hard to anticipate World War II, hormonal birth control, and AIDS, amongst other things.

Typhoid counterfactuals

An acquaintance (who doesn't work in public health) recently got typhoid while traveling. She noted that she had had the typhoid vaccine less than a year ago but got sick anyway. Surprisingly to me, even though she knew "the vaccine was only about 50% effective" she now felt that it was  a mistake to have gotten the vaccine. Why? "If you're going to get the vaccine and still get typhoid, what's the point?" I disagreed but am afraid my defense wasn't particularly eloquent in the moment: I tried to say that, well, if it's 50% effective and you and, I both got the vaccine, then only one of us would get typhoid instead of both of us. That's better, right? You just drew the short straw. Or, if you would have otherwise gotten typhoid twice, now you'll only get it once!

These answers weren't reassuring in part because thinking counterfactually -- what I was trying to do -- isn't always easy. Epidemiologists do this because they're typically told ad nauseum to approach causal questions by first thinking "how could I observe the counterfactual?" At one point after finishing my epidemiology coursework I started writing a post called "The Top 10 Things You'll Learn in Public Health Grad School" and three or four of the ten were going to be "think counterfactually!"

A particularly artificial and clean way of observing this difference -- between what happened and what could have otherwise happened -- is to randomly assign people to two groups (say, vaccine and placebo). If the groups are big enough to average out any differences between them, then the differences in sickness you observe are due to the vaccine. It's more complicated in practice, but that's where we get numbers like the efficacy of the typhoid vaccine -- which is actually a bit higher than 50%.

You can probably see where this is going: while the randomized trial gives you the average effect, for any given individual in the trial they might or might not get sick. Then, because any individual is assigned only to the treatment or control, it's hard to pin their outcome (sick vs. not sick) on that alone. It's often impossible to get an exhaustive picture of individual risk factors and exposures so as to explain exactly which individuals will get sick or not in advance. All you get is an average, and while the average effect is really, really important, it's not everything.

This is related somewhat to Andrew Gelman's recent distinction between forward and reverse causal questions, which he defines as follows:

1. Forward causal inference. What might happen if we do X? What are the effects of smoking on health, the effects of schooling on knowledge, the effect of campaigns on election outcomes, and so forth?

2. Reverse causal inference. What causes Y? Why do more attractive people earn more money? Why do many poor people vote for Republicans and rich people vote for Democrats? Why did the economy collapse?

The randomized trial tries to give us an estimate of the forward causal question. But for someone who already got sick, the reverse causal question is primary, and the answer that "you were 50% less likely to have gotten sick" is hard to internalize. As Gelman says:

But reverse causal questions are important too. They’re a natural way to think (consider the importance of the word “Why”) and are arguably more important than forward questions. In many ways, it is the reverse causal questions that lead to the experiments and observational studies that we use to answer the forward questions.

The moral of the story -- other than not sharing your disease history with a causal inference buff -- is that reconciling the quantitative, average answers we get from the forward questions with the individual experience won't always be intuitive.

African population density

I was recently struck by differences in population density: Northern Nigeria's Kano state has an official population of ~10 million, whereas the entire country of Zambia has 13.5. Zambia's land area, meanwhile, is also about 35 times that of Kano. So I started looking around for a nice map of population density in Africa. The best I found was this one via UNEP:

And here's a higher resolution version.

Some of the most striking concentrations are along the Mediterranean coast, the Nile basin, the Ethiopian plateau, and around Lake Victoria. (I'd love to track down the data behind this map but haven't had time.)

A good map can change how you think. If you're used to seeing maps that have country-level estimates of disease prevalence, for instance, you miss variations at the subnational level. This is often for good reason, as the subnational data is often even spottier than the national estimates. But another thing you miss is a sense of absolute population numbers, because looking at a map it's much easier to see countries by their areas rather than their populations, which for matters of health and other measures of human well-being is generally what we care about. There are some cool maps that do this but they inevitably lose their geographic accuracy.

"When public health works, it's invisible"

Caitlin Rivers' post on the "public health paradox: why people don't get flu shots" hits the nail on the head:

Unfortunately, the root of this problem is deep. The problem is that when public health works, it is invisible. It's an insidious, persistent public relations issue that plagues public health. Nobody sees when a chain of disease transmission is broken, or when contaminated food is prevented from reaching the market, or when toxic pollutants don't enter the environment. That's the point: the goal of public health is prevention, not reaction....

What then can be done to counteract these misperceptions? First, public health needs to be more vocal about its successes. This graphic of crude death rates for infectious diseases during the 19th century, for example, should be widely disseminated. A little self-promotion could go a long ways.

That's one reason I like Millions Saved, from the Center for Global Development -- it highlights "proven success in global health." One of the things that struck me when reading it was that most of the people who benefited from these interventions and programs would have no way of knowing that they benefited.

For another positive take, check out Charles Kenny's book Getting Better.

An uphill battle

I took this photo in the NYC subway a few days ago. My apologies for the quality, but I thought it's a great juxtaposition:

In the top of the photo is an ad from the NYC Department of Health, advising you to choose food with less sodium. (Here's an AP story about the ads.) But to the bottom right is an ad for McDonald's dollar menu, and those are everywhere. While it doesn't mean we shouldn't run such ads, it's worth remembering that the sheer volume of food advertising will always dwarf opposing health messages. 

Rearranging the malarial deck chairs?

A friend sent this link to me, highlighting a critical comment about the future of the World Health Organization, in the context of the World Malaria Report 2012. Here's an excerpt of the comment by William Jobin:

Their 2012 Annual Report is a very disturbing report from WHO, for at least two reasons:

1. Their program is gradually falling apart, and they offer no way to refocus, no strategy for dealing with the loss in funding, nor the brick wall of drug and biocide resistance which is just down the road. There is a label for people who keep doing the same thing, but expect different results. Do you remember what it is?

2. Because the entire top management of WHO consists of physicians, they have no idea of the opportunities they are missing for additional funding and for additional methods to add to their chemically-oriented strategy...

Concluding with:

I am not sure WHO has much of a future, nor does the UN system itself, after their failure to prevent the wars in Libya and Syria. But as long as the UN and WHO continue to operate, they must refocus their approach to face the reality of a rapidly declining budget from UN sources. Instead, I see them just re-arranging the deck chairs on the Titanic.

My friend said, "I wish these comments (and issues with the WHO and UN) were more publicised! This is not the first time I am hearing of such issues with the WHO and its demise." I've certainly heard similar sentiments about the WHO from classmates and professors, but it seems there's much less open discussion than you might expect. I'd welcome discussion in the comments...

This beautiful graphic is not really that useful

This beautiful infographic from the excellent blog Information is Beautiful has been making the rounds. You can see a bigger version here, and it's worth poking around for a bit. The creators take all deaths from the 20th century (drawing from several sources) and represent their relative contribution with circles:

I appreciate their footnote that says the graphic has "some inevitable double-counting, broad estimation and ball-park figures." That's certainly true, but the inevitably approximate nature of these numbers isn't my beef.

The problem is that I don't think raw numbers of deaths tell us very much, and can actually be quite misleading. Someone who saw only this infographic might well end up less well-informed than if they didn't see it. Looking at the red circles you get the impression that non-communicable and infectious diseases were roughly equivalent in importance in the 20th century, followed by "humanity" (war, murder, etc) and cancer.

The root problem is that mortality is inevitable for everyone, everywhere. This graphic lumps together pneumonia deaths at age 1 with car accidents at age 20, and cancer deaths at 50 with heart disease deaths at 80. We typically don't  (and I would argue should't) assign the same weight to a death in childhood or the prime of life with one that comes at the end of a long, satisfying life.  The end result is that this graphic greatly overemphasizes the importance of non-communicable diseases in the 20th century -- that's the impression most laypeople will walk away with.

A more useful graphic might use the same circles to show the years of life lost (or something like DALYs or QALYs) because those get a bit closer at what we care about. No single number is actually  all that great, so we can get a better understanding if we look at several different outcomes (which is one problem with any visualization). But I think raw mortality numbers are particularly misleading.

To be fair, this graphic was commissioned by Wellcome as "artwork" for a London exhibition, so maybe it should be judged by a different standard...

First responses to DEVTA roll in

In my last post I highlighted the findings from the DEVTA trial of deworming in Vitamin A in India, noting that the Vitamin A results would be more controversial. I said I expected commentaries over the coming months, but we didn't have to wait that long after all. First is a BBC Health Check program features a discussion of DEVTA with Richard Peto, one of the study's authors. It's for a general audience so it doesn't get very technical, and because of that it really grated when they described this as a "clinical trial," as that has certain connotations of rigor that aren't reflected in the design of the study. If DEVTA is a clinical trial, then so was

Peto also says there were two reasons for the massive delay in publishing the trial, 1) time to check things and "get it straight," and 2) that they were " afraid of putting up a trial with a false negative." [An aside for those interested in publication bias issues: can you imagine an author with strong positive findings ever saying the same thing about avoiding false positives?!]

Peto ends by sounding fairly neutral re: Vitamin A (portraying himself in a middle position between advocates in favor and skeptics opposed) but acknowledges that with their meta-analysis results Vitamin A is still "cost-effective by many criteria."

Second is a commentary in The Lancet by Al Sommers, Keith West, and Reynaldo Martorell. A little history: Sommers ran the first big Vitamin A trials in Sumtra (published in 1986) and is the former dean of the Johns Hopkins School of Public Health.  (Sommers' long-term friendship with Michael Bloomberg, who went to Hopkins as an undergrad, is also one reason the latter is so big on public health.) For more background, here's a recent JHU story on Sommers' receiving a $1 million research prize in part for his work on Vitamin A.

Part of their commentary is excerpted below, with my highlights in bold:

But this was neither a rigorously conducted nor acceptably executed efficacy trial: children were not enumerated, consented, formally enrolled, or carefully followed up for vital events, which is the reason there is no CONSORT diagram. Coverage was ascertained from logbooks of overworked government community workers (anganwadi workers), and verified by a small number of supervisors who periodically visited randomly selected anganwadi workers to question and examine children who these workers gathered for them. Both anganwadi worker self-reports, and the validation procedures, are fraught with potential bias that would inflate the actual coverage.

To achieve 96% coverage in Uttar Pradesh in children found in the anganwadi workers' registries would have been an astonishing feat; covering 72% of children not found in the anganwadi workers' registries seems even more improbable. In 2005—06, shortly after DEVTA ended, only 6·1% of children aged 6—59 months in Uttar Pradesh were reported to have received a vitamin A supplement in the previous 6 months according to results from the National Family Health Survey, a national household survey representative at national and state level.... Thus, it is hard to understand how DEVTA ramped up coverage to extremely high levels (and if it did, why so little of this effort was sustained). DEVTA provided the anganwadi workers with less than half a day's training and minimal if any incentive.

They also note that the study funding was minimalist compared to more rigorous studies, which may be an indication of quality. And as an indication that there will almost certainly be alternative meta-analyses that weight the different studies differently:

We are also concerned that Awasthi and colleagues included the results from this study, which is really a programme evaluation, in a meta-analysis in which all of the positive studies were rigorously designed and conducted efficacy trials and thus represented a much higher level of evidence. Compounding the problem, Awasthi and colleagues used a fixed-effects analytical model, which dramatically overweights the results of their negative findings from a single population setting. The size of a study says nothing about the quality of its data or the generalisability of its findings.

I'm sure there will be more commentaries to follow. In my previous post I noted that I'm still trying to wrap my head around the findings, and I think that's still right. If I had time I'd dig into this a bit more, especially the relationship with the Indian National Family Health Survey. But for now I think it's safe to say that two parsimonious explanations for how to reconcile DEVTA with the prior research are emerging:

1. DEVTA wasn't all that rigorous and thus never achieved the high population coverage levels necessary to have a strong mortality impact; the mortality impact was attenuated by poor coverage, resulting in the lack of a statistically significant effect in line with prior results. Thus is shouldn't move our priors all that much. (Sommers et al. seem to be arguing for this.) Or,

2. There's some underlying change in the populations between the older studies and these newer studies that causes the effect of Vitamin A to decline -- this could be nutrition, vaccination status, shifting causes of mortality, etc. If you believe this, then you might discount studies because they're older.

(h/t to @karengrepin for the Lancet commentary.)

A massive trial, a huge publication delay, and enormous questions

It's been called the "largest clinical* trial ever": DEVTA (Deworming and Enhanced ViTamin A supplementation), a study of Vitamin A supplementation and deworming in over 2 million children in India, just published its results. "DEVTA" may mean "deity" or "divine being" in Hindi but some global health experts and advocates will probably think these results come straight from the devil. Why? Because they call into question -- or at least attenuate -- our estimates of the effectiveness of some of the easiest, best "bang for the buck" interventions out there. Data collection was completed in 2006, but the results were just published in The Lancet. Why the massive delay? According to the accompany discussion paper, it sounds like the delay was rooted in very strong resistance to the results after preliminary outcomes were presented at a conference in 2007. If it weren't for the repeated and very public shaming by the authors of recent Cochrane Collaboration reviews, we might not have the results even today. (Bravo again, Cochrane.)

So, about DEVTA. In short, this was a randomized 2x2 factorial trial, like so:

The results were published as two separate papers, one on Vitamin A and one on deworming, with an additional commentary piece:

• "Vitamin A supplementation every 6 months with retinol in 1 million pre-school children in north India: DEVTA, a cluster-randomised trial"
• "Population deworming every 6 months with albendazole in 1 million pre-school children in north India: DEVTA, a cluster-randomised trial"
• Commentary: "DEVTA: results from the biggest clinical trial ever"

The controversy is going to be more about what this trial didn't find, rather than what they did: the confidence interval on the Vitamin A study's mortality estimate (mortality ratio 0.96, 95% confidence interval of 0.89 to 1.03) is consistent with a mortality reduction as large as 11%, or as much as a 3% increase. The consensus from previous Vitamin A studies was mortality reductions of 20-30%, so this is a big surprise. Here's the abstract to that paper:

Background

In north India, vitamin A deficiency (retinol <0·70 μmol/L) is common in pre-school children and 2–3% die at ages 1·0–6·0 years. We aimed to assess whether periodic vitamin A supplementation could reduce this mortality.

Methods

Participants in this cluster-randomised trial were pre-school children in the defined catchment areas of 8338 state-staffed village child-care centres (under-5 population 1 million) in 72 administrative blocks. Groups of four neighbouring blocks (clusters) were cluster-randomly allocated in Oxford, UK, between 6-monthly vitamin A (retinol capsule of 200 000 IU retinyl acetate in oil, to be cut and dripped into the child’s mouth every 6 months), albendazole (400 mg tablet every 6 months), both, or neither (open control). Analyses of retinol effects are by block (36 vs36 clusters).

The study spanned 5 calendar years, with 11 6-monthly mass-treatment days for all children then aged 6–72 months.  Annually, one centre per block was randomly selected and visited by a study team 1–5 months after any trial vitamin A to sample blood (for retinol assay, technically reliable only after mid-study), examine eyes, and interview caregivers. Separately, all 8338 centres were visited every 6 months to monitor pre-school deaths (100 000 visits, 25 000 deaths at ages 1·0–6·0 years [the primary outcome]). This trial is registered at ClinicalTrials.gov, NCT00222547.

Findings

Estimated compliance with 6-monthly retinol supplements was 86%. Among 2581 versus 2584 children surveyed during the second half of the study, mean plasma retinol was one-sixth higher (0·72 [SE 0·01] vs 0·62 [0·01] μmol/L, increase 0·10 [SE 0·01] μmol/L) and the prevalence of severe deficiency was halved (retinol <0·35 μmol/L 6% vs13%, decrease 7% [SE 1%]), as was that of Bitot’s spots (1·4% vs3·5%, decrease 2·1% [SE 0·7%]).

Comparing the 36 retinol-allocated versus 36 control blocks in analyses of the primary outcome, deaths per child-care centre at ages 1·0–6·0 years during the 5-year study were 3·01 retinol versus 3·15 control (absolute reduction 0·14 [SE 0·11], mortality ratio 0·96, 95% CI 0·89–1·03, p=0·22), suggesting absolute risks of death between ages 1·0 and 6·0 years of approximately 2·5% retinol versus 2·6% control. No specific cause of death was significantly affected.

Interpretation

DEVTA contradicts the expectation from other trials that vitamin A supplementation would reduce child mortality by 20–30%, but cannot rule out some more modest effect. Meta-analysis of DEVTA plus eight previous randomised trials of supplementation (in various different populations) yielded a weighted average mortality reduction of 11% (95% CI 5–16, p=0·00015), reliably contradicting the hypothesis of no effect.

Note that instead of just publishing these no-effect results and leaving the meta-analysis to a separate publication, the authors go ahead and do their own meta-analysis of DEVTA plus previous studies and report that -- much attenuated, but still positive -- effect in their conclusion. I think that's a fair approach, but also reveals that the study's authors very much believe there are large Vitamin A mortality effects despite the outcome of their own study!

[The only media coverage I've seen of these results so far comes from the Times of India, which includes quotes from the authors and Abhijit Banerjee.]

To be honest, I don't know what to make of the inconsistency between these findings and previous studies, and am writing this post in part to see what discussion it generates. I imagine there will be more commentaries on these findings over the coming months, with some decrying the results and methodologies and others seeing vindication in them. In my view the best possible outcome is an ongoing concern for issues of external validity in biomedical trials.

What do I mean? Epidemiologists tend to think that external validity is less of an issue in randomized trials of biomedical interventions -- as opposed to behavioral, social, or organizational trials -- but this isn't necessarily the case. Trials of vaccine efficacy have shown quite different efficacy for the same vaccine (see BCG and rotavirus) in different locations, possibly due to differing underlying nutritional status or disease burdens. Our ability to interpret discrepant findings can only be as sophisticated as the available data allows, or as sophisticated as allowed by our understanding of the biological and epidemiologic mechanisms that matter on the pathway from intervention to outcome. We can't go back in time and collect additional information (think nutrition, immune response, baseline mortality, and so forth) on studies far in the past, but we can keep such issues in mind when designing trials moving forward.

All that to say, these results are confusing, and I look forward to seeing the global health community sort through them. Also, while the outcomes here (health outcomes) are different from those in the Kremer deworming study (education outcomes), I've argued before that lack of effect or small effects on the health side should certainly influence our judgment of the potential education outcomes of deworming.

*I think given the design it's not that helpful to call this a 'clinical' trial at all - but that's another story.