« Polls Suggest Possibility Of Big Obama New Hampshire Win
A View from Israel: US Presidential Candidates & the Middle East »
The Civilian Body Count Controversy & Other News From The Forever War
Ascertaining an accurate count of the number of civilians to die in the Iraq war — a bloodbath that they neither invited nor deserved — has been impossible. Some deaths are never reported, some are suppressed and the people who keep track of the carnage often have an ax to grind, which results in a predisposition to under or over report.
If there is anything approaching a consensus view, and I use that term advisedly, it is that somewhere between 80,000 and 87,000 civilians have died since the U.S.-led invasion in March 2001. That is the range arrived at by the folks at Iraq Body County based on data cross-checked from media reports and hospitals, morgues, non-government organizations and official figures.
That number stands in stark contrast to the claim that 601,027 civilians died in a Johns Hopkins University cluster-sample survey published three weeks before the mid-term elections in The Lancet. By contrast, Iraq Body Count listed 47,702 deaths during the same period.
The survey provoked a firestorm. Anti-war advocates seized upon it to advance their view that the civilian toll was far worse than was being acknowledged, while pro-war advocates argued that it was deeply flawed.
Drawing on my own experience in survey taking and analysis and despite a healthy dose of skepticism, I myself came down somewhat on the side of the survey. But just as there still are people looking for proof of those elusive Iraqi WMD, there have been people hard at work trying to debunk the survey.
“Data Bomb,” the most thorough and persuasive effort to date, has just been published in The National Journal.
Authors Neil Munro and Carl M. Cannon write that the survey authors:
* Were ideologically predisposed to conclude that there was a much higher body count and the timing of the survey’s release just before the election was no accident.
* Followed a model that ensured that even minor components of the data, when extrapolated over the whole population, would yield substantially higher numbers.
* Have made it difficult to resolve apparent inconsistencies in their methodology and analysis by not making available to other researchers the surveyors’ original field reports and response forms and not just collated survey results.
* May have engaged in fraud.
The survey was led Gilbert Burnham, a Johns Hopkins University professor, with assistance from Riyadh Lafta, and Les Roberts.
Burnham defends the survey, although not all that persuasively, in an interview with Pajamas Media editor Richard Miniter.
Please click here to read more at Kiko’s House.
14 Responses to “The Civilian Body Count Controversy & Other News From The Forever War”
[...] Woody Maxim wrote an interesting post today onHere’s a quick excerptThat number stands in stark contrast to the claim that 601027 civilians died in a Johns Hopkins University cluster-sample survey published three weeks before the mid-term elections in The Lancet. By contrast, Iraq Body Count listed … [...]
Shaun – The National Journal piece is just as guilty of fraud and bias as the authors claim the Lancet study was.
One example:(NJ article)
A response from Kane, at his blog related to the Lancet study:
Kane is saying that Munro is taking him out of context.
Rudi, I don't care what the National Journal's biases are if they accurately poke holes in the methodology of the study. The burden is on the study authors to show proper scientific scholarship.
Fraud, however, is another matter… Let's see how this plays out over the next couple days/weeks as this heats up.
Both Rudi and Idiosyncrat make valid points. Of course the National Journal has its own biases, but it does point out in some depth what would seem to be serious problems with the study.
For what it may be worth, I don't typically quote from the NJ, but because I had gone on record more or less buying into the study, I believed that it was important to write about the NJ story.
The first criticism(1) is just plain absurd, a survey is not an experiment. The scientific method really doesn't apply to studies. Question the methods and statistics, but WTF does an experiment have to do with the Lancet study. The study authors didn't claim that “cold fusion” is possible.
The second (2) criticism is plausible but very weak. Discount a study because it was running behind schedule in war torn Iraq. The “main street bias” argument is plausible, but a time delay/schedule is stupid.
One other point, Munro uses the IBC numbers as a sly way to discount the Lancet study. But, IBC isn't scientific and Iraq's Interior ministry came out with a figure of 150.000 dead. A four to one discrepancy isn't as bad as 10 to one. And bringing up Soros is a red herring…
Shaun, I don't blame you for giving them the benefit of the doubt. The Lancet isn't some fly-by-night publication… It's one of the most respected peer-reviewed medical journals on the planet. Sure, one doesn't have to dig all that deep to find academics performing questionable scholarship, but getting one's work published in The Lancet is another story. Or at least it's supposed to be. Not unprecedented for them to get duped every now and then, but it's rare and usually involves serious fraud on the part of the researchers.
Criticism like this happens all the time in academia and is usually ignored by a general public who would be bored to tears by such discourse. And amongst those who aren't playing the publish-or-perish game, you'll see attacking the very methodologies accepted by the academic community. Like with the topic of global warming, in this particular case, it's just playing out more publicly because of topic.
Let's see if the challenge stands up.
Rudi, I'm not a fan of the “experiment” language they used, but it's a red herring. Being able to replicate findings is indeed very important when doing these studies. In many environments, academics must keep their original raw data on hand for x number of years in the case of challenges. It's considered a norm to share such data when requested, and if your research is in any way supposed by state or federal funds, i believe that the researcher is required to make that data available since the public technically owns it.
In terms of your second point, this is where my brain starts to short-circuit. Who should be counted, when and how is verrry subjective. That why we have “experts” who figure these things out… My guess is that this is where we'll see the resulting huge gap in numbers and allegations of political tampering that may unfortunately be impossible to resolve objectively.
I don't find Burnham's defense so weak. For example, Pajamas Media criticizes the high number of reported suicide bomber victims, because such bombings are generally well reported. Maybe some of these victims were actually killed by IEDs or collateral damage from other ordinance, but with 80% of the deaths confirmed by death certificates, the death count would seem to be strong, whether or not the cause is. Discount the 655,000 number by 20% and you have 100% confirmation by death certificate. That's still 524,000 deaths, a devastating toll.
And the challenges to the Saddam-era death rate, while interesting to contemplate, are in line with other estimates, as Burnham notes. It's a pretty young population, so not inconceivable. But the key point is that the death of a family member is a very finite endpoint. In almost all cases, the family would be hard pressed to claim a nonexistent death or to underestimate the number of still-living family members, especially with confirmation by death certificate.
The most legitimate criticism of the study is actually offered by one of the commenters, that the study used to few clusters, possibly only 1/5 as many as needed to legitimize the figures. Fine. Let's look at a bunch more clusters, rather than sweeping the whole issue under the rug as if it's a political talking point. Maybe it is. Maybe it isn't. More research can determine that.
However, this research was based on face-to-face interviews with 1,880 families. We regularly cite, use and trust surveys of far fewer individuals in applications from political polling to product development to entertainment ratings.
Ids – On the point of releasing data, Munro is wrong in that the raw data was released last February.
The second issue is also addressed in this same article.
http://intl.emboj.org/nature/journal/v446/n7131…
From the “Data bomb” article:
Munro claims that the data isn't made available, this is an outright LIE.
Here are some links to prove the data was available last year.
http://lancetiraq.blogspot.com/2007/04/data.html
http://lancetiraq.blogspot.com/2007/04/data-now…
http://www.jhsph.edu/refugee/research/iraq/#Rel…
Rudi, well, in that case, Munro just stuck a big fat boot (army, cowboy, mountaineering or Hugo Boss — your choice) in his mouth. Do you know if anyone has done anything with that data yet?
No, but I found this by just Googling the quoted authors. Munro IMHO would have direct contact with said authors, for him to lie is journalistic fraud. Wasn't that one of his claims…
I will look into the data analysis, here is another web site by academics involved in the spat.
http://www.rhul.ac.uk/economics/Research/confli…
“Munro claims that the data isn't made available, this is an outright LIE.”
I think you're seeing what you want to see here Rudi. You don't seem to be paying attention. The article even says:
“Under pressure from critics, the authors did release a disk of the surveyors' collated data, including tables showing how often the survey teams said they requested to see, and saw, the death certificates. But those tables are suspicious, in part, because they show data-heaping, critics said.”
That disk is the data you're talking about. Certainly the NJ article is not hiding that this data has been released. There's even a sidebar illustrating a piece of that released data (which btw, seems to be suggestive of fraud):
http://nationaljournal.com/njcover.htm#
The passage you quote is clearly referring to other aspects of the data and forms which have been kept secret, not the same as the data disk that was released. You gloss over the particulars here with a generalized “the data” in order to call Munro a liar, as if “the data” covers everything in question. It doesn't.
And if you want to go there, it's really Burnham who, if anyone, should be called on outright lies for his claims in the Pajamas Media piece. He claims for example that with the 2004 and 2006 studies:
“The methodology was the same, though we increased the sample size and the number of clusters [from 33 to 47].”
So supposedly they were exactly the same, but the 2006 one was just larger (and therefore supposedly better). Actually, the methodology was quite different in 2006 (and more suspect). The pages on the link that Rudi posts above to rhul actually have many pages on this and discuss serious problems with the 2006 methodology that don't directly apply to the 2004 methodology: because they were different. The 2004 one says it used GPS units to select each house, not his “Main Street” thing that the 2006 one did.
Then Burnham even claims that “The 2006 data from January 2002 through Sept 2004 (end of the 2004 study) were virtually identical to the 2004 studies.”
Apparently this should allay concerns: The methodologies were the same and the results were “virtually identical” In fact, the same website debunks this claim (outright LIE?) in detail here:
http://www.rhul.ac.uk/economics/Research/confli…
The two studies actually show very different things for the same period. You can even see this in the NJ article graph on the right side. The 2006 one looks way out of whack.
sonny – I was referring to the implied claim that full data wasn't available to peer scientists, not partisans at Memoramum. In the other comment I linked to the actual virtual form(for the data) and David Kane's Blog dealing with Lancet study issues. Here is the complete post in regards to the availability of the data.