The Factinista

There has of late been a great deal of criticism, particularly from the Bush administration, the Pentagon, and the Iraqi government, of a recent study of "mortality after the 2003 invasion of Iraq" published in The Lancet, a highly respected British medical journal. But in a recent post on the Guardian blog, "Comment is Free," Daniel Davies explains why "the numbers do add up." Davies makes it very clear that the main point of the study is not to provide a precise death toll but to warrant qualitative conclusions about the effects of the Iraq war.

First, don't concentrate on the number 600,000 (or 655,000, depending on where you read). This is a point estimate of the number of excess Iraqi deaths - it's basically equal to the change in the death rate since the invasion, multiplied by the population of Iraq, multiplied by three-and-a-quarter years. Point estimates are almost never the important results of statistical studies and I wish the statistics profession would stop printing them as headlines.

The question that this study was set up to answer was: as a result of the invasion, have things got better or worse in Iraq? And if they have got worse, have they got a little bit worse or a lot worse. Point estimates are only interesting in so far as they demonstrate or dramatise the answer to this question.

The results speak for themselves. There was a sample of 12,801 individuals in 1,849 households, in 47 geographical locations. That is a big sample, not a small one. The opinion polls from Mori and such which measure political support use a sample size of about 2,000 individuals, and they have a margin of error of +/- 3%. If Margaret Beckett looks at the Labour party's rating in the polls, she presumably considers this to be reasonably reliable, so she should not contribute to public ignorance by allowing her department to disparage "small samples extrapolated to the whole country". The Iraq Body Count website and the Iraqi government statistics are not better measures than the survey results, because one of the things we know about war zones is that casualties are under-reported, usually by a factor of more than five.

And the results were shocking. In the 18 months before the invasion, the sample reported 82 deaths, two of them from violence. In the 39 months since the invasion, the sample households had seen 547 deaths, 300 of them from violence. The death rate expressed as deaths per 1,000 per year had gone up from 5.5 to 13.3.

Talk of confidence intervals becomes frankly irrelevant at this point. If you want to pick a figure for the precise number of excess deaths, then (1.33% - 0.55%) x 26,000,000 x 3.25 = 659,000 is as good as any, multiplying out the difference between the death rates by the population of Iraq and the time since the invasion. But we're interested in the qualitative conclusion here.

That qualitative conclusion is this: things have got worse, and they have got a lot worse, not a little bit worse. Whatever detailed criticisms one might make of the methodology of the study (and I have searched assiduously for the last two years, with the assistance of a lot of partisans of the Iraq war who have tried to pick holes in the study, and not found any), the numbers are too big. If you go out and ask 12,000 people whether a family member has died and get reports of 300 deaths from violence, then that is not consistent with there being only 60,000 deaths from violence in a country of 26 million. It is not even nearly consistent.

All right, so Davies isn't a statistician by trade. But here are favorable comments on the Lancet article by professional statisticians:

"Given the conditions (in Iraq), it's actually quite a remarkable effort," said Steve Heeringa, director of the statistical design group at the Institute for Social Research at the University of Michigan.

"I can't imagine them doing much more in a much more rigorous fashion."

He said the study made "minor departures" from the standards generally used in national surveys for choosing what households to interview. Whether those departures, brought on by wartime conditions in Iraq, introduced a bias in the results is impossible to measure from the data alone, he said.

Frank Harrell Jr., chair of the biostatistics department at Vanderbilt University, called the study design solid and said it included "rigorous, well-justified analysis of the data."

And Richard Brennan, head of health programs at the New York-Based International Rescue Committee, said the study's survey approach was typical.

"This is the most practical and appropriate methodology for sampling that we have in humanitarian conflict zones," said Brennan, whose group has conducted similar projects in Kosovo, Uganda and Congo.

"While the results of this survey may startle people, it's hard to argue with the methodology at this point."

Donald Berry, chairman of the statistics department at the University of Texas' M.D. Anderson Cancer Center in Houston, said he believes the study was done "in a reasonable way." But he said the range of uncertainty given for the estimates was much too narrow, because of potential statistical biases in the survey.

While it is impossible to calculate a better range that accounts for that, he said, it would not be surprising if the low end dropped about four-fold to 100,000 deaths. A wider range of uncertainty would make the 655,000 figure less meaningful, he said.

Fair enough. Maybe it's only 100,000 deaths at the low end. A mere 100,000 deaths, if such a locution is morally permissible in these circumstances. But again the main point is not to deliver an accurate death toll, rather to make it clear that in fact things have gotten much, much worse in Iraq since the 2003 invasion. As if this wasn't already clear enough, we now have sound statistical evidence for this conclusion. And yet, Bush:

"I don't consider it a credible report," Bush said Wednesday.

Neither does Gen. George W. Casey, the top American military commander in Iraq.

"That 650,000 number seems way, way beyond any number that I have seen," Casey said. "I've not seen a number higher than 50,000. And so I don't give it that much credibility at all."

And neither does Michael E. O'Hanlon of the Brookings Institution, which also tracks Iraqi deaths.

"I do not believe the new numbers. I think they're way off," he said.

Other research methods on the ground, like body counts, forensic analysis and taking eyewitness reports, have produced numbers only about one-tenth as high, he said. "I have a hard time seeing how all the direct evidence could be that far off ... therefore I think the survey data is probably what's wrong."

Of course, none of these comments are unexpected. But at the same time it never fails to astonish me how, when dealing with, say, information from an apparently highly reliable source about something as serious as the deaths of perhaps hundreds of thousands of civilians, the president of the United States allows himself the luxury of being dismissive, and the political cost is nil. What underlies this is perhaps the equally astonishing fact that Americans at large are tremendously suspicious of science.
Then again, we're Americans -- so, as Steven Colbert once said to President Bush, "We're not brainiacs on the nerd patrol. We're not members of the factinista. We go straight from the gut, right sir? That's where the truth lies, right down here in the gut. Do you know you have more nerve endings in your gut than you have in your head? You can look it up. I know some of you are going to say 'I did look it up, and that's not true.' That's 'cause you looked it up in a book."