« Polls Suggest Possibility Of Big Obama New Hampshire Win
A View from Israel: US Presidential Candidates & the Middle East »



The Civilian Body Count Controversy & Other News From The Forever War

January 7th, 2008
By SHAUN MULLEN, TMV Columnist

Print Print

01abaghdad_street.0.jpg

Ascertaining an accurate count of the number of civilians to die in the Iraq war — a bloodbath that they neither invited nor deserved — has been impossible. Some deaths are never reported, some are suppressed and the people who keep track of the carnage often have an ax to grind, which results in a predisposition to under or over report.

If there is anything approaching a consensus view, and I use that term advisedly, it is that somewhere between 80,000 and 87,000 civilians have died since the U.S.-led invasion in March 2001. That is the range arrived at by the folks at Iraq Body County based on data cross-checked from media reports and hospitals, morgues, non-government organizations and official figures.

That number stands in stark contrast to the claim that 601,027 civilians died in a Johns Hopkins University cluster-sample survey published three weeks before the mid-term elections in The Lancet. By contrast, Iraq Body Count listed 47,702 deaths during the same period.

The survey provoked a firestorm. Anti-war advocates seized upon it to advance their view that the civilian toll was far worse than was being acknowledged, while pro-war advocates argued that it was deeply flawed.

Drawing on my own experience in survey taking and analysis and despite a healthy dose of skepticism, I myself came down somewhat on the side of the survey. But just as there still are people looking for proof of those elusive Iraqi WMD, there have been people hard at work trying to debunk the survey.

“Data Bomb,” the most thorough and persuasive effort to date, has just been published in The National Journal.

Authors Neil Munro and Carl M. Cannon write that the survey authors:

* Were ideologically predisposed to conclude that there was a much higher body count and the timing of the survey’s release just before the election was no accident.

* Followed a model that ensured that even minor components of the data, when extrapolated over the whole population, would yield substantially higher numbers.

* Have made it difficult to resolve apparent inconsistencies in their methodology and analysis by not making available to other researchers the surveyors’ original field reports and response forms and not just collated survey results.

* May have engaged in fraud.

The survey was led Gilbert Burnham, a Johns Hopkins University professor, with assistance from Riyadh Lafta, and Les Roberts.

Burnham defends the survey, although not all that persuasively, in an interview with Pajamas Media editor Richard Miniter.

Please click here to read more at Kiko’s House.



This entry was posted on Monday, January 7th, 2008 at 4:42 am and is filed under Iraq. You can leave a response, or trackback from your own site.

Viewing 13 Comments

    • ^
    • v
    Shaun - The National Journal piece is just as guilty of fraud and bias as the authors claim the Lancet study was.

    One example:(NJ article)
    According to a data table reviewed by Spagat and Kane, the team recorded the violent deaths as taking place in early July and did not explain why they failed to see death certificates for any of the 24 victims. The surveyors did remember, however, to ask for the death certificate of the one person who had died peacefully in that cluster.


    A response from Kane, at his blog related to the Lancet study:
    Lancetiraq.blogspot.com/
    Neil Munro's National Journal article is out. I haven't had a chance to read it closely, but my quotes are not as contextualized as I would like them to be.

    Kane is saying that Munro is taking him out of context.
    • ^
    • v
    Rudi, I don't care what the National Journal's biases are if they accurately poke holes in the methodology of the study. The burden is on the study authors to show proper scientific scholarship.

    Fraud, however, is another matter... Let's see how this plays out over the next couple days/weeks as this heats up.
    • ^
    • v
    Both Rudi and Idiosyncrat make valid points. Of course the National Journal has its own biases, but it does point out in some depth what would seem to be serious problems with the study.

    For what it may be worth, I don't typically quote from the NJ, but because I had gone on record more or less buying into the study, I believed that it was important to write about the NJ story.
    • ^
    • v
    (1)Still, the authors have declined to provide the surveyors' reports and forms that might bolster confidence in their findings. Customary scientific practice holds that an experiment must be transparent -- and repeatable -- to win credence. Submitting to that scientific method, the authors would make the unvarnished data available for inspection by other researchers. Because they did not do this, citing concerns about the security of the questioners and respondents, critics have raised the most basic question about this research: Was it verifiably undertaken as described in the two Lancet articles?
    ...

    (2)The authors should not have included the July data in their report because the survey was scheduled to end on June 30, according to Debarati Guha-Sapir, director of the World Health Organization's Collaborating Center for Research on the Epidemiology of Disasters at the University of Louvain in Belgium. Because of the study's methodology, those 24 deaths ultimately added 48,000 to the national death toll and tripled the authors' estimate for total car bomb deaths to 76,000. That figure is 15 times the 5,046 car bomb killings that Iraq Body Count recorded up to August 2006.


    The first criticism(1) is just plain absurd, a survey is not an experiment. The scientific method really doesn't apply to studies. Question the methods and statistics, but WTF does an experiment have to do with the Lancet study. The study authors didn't claim that "cold fusion" is possible.

    The second (2) criticism is plausible but very weak. Discount a study because it was running behind schedule in war torn Iraq. The "main street bias" argument is plausible, but a time delay/schedule is stupid.

    One other point, Munro uses the IBC numbers as a sly way to discount the Lancet study. But, IBC isn't scientific and Iraq's Interior ministry came out with a figure of 150.000 dead. A four to one discrepancy isn't as bad as 10 to one. And bringing up Soros is a red herring...
    • ^
    • v
    Shaun, I don't blame you for giving them the benefit of the doubt. The Lancet isn't some fly-by-night publication... It's one of the most respected peer-reviewed medical journals on the planet. Sure, one doesn't have to dig all that deep to find academics performing questionable scholarship, but getting one's work published in The Lancet is another story. Or at least it's supposed to be. Not unprecedented for them to get duped every now and then, but it's rare and usually involves serious fraud on the part of the researchers.

    Criticism like this happens all the time in academia and is usually ignored by a general public who would be bored to tears by such discourse. And amongst those who aren't playing the publish-or-perish game, you'll see attacking the very methodologies accepted by the academic community. Like with the topic of global warming, in this particular case, it's just playing out more publicly because of topic.

    Let's see if the challenge stands up.
    • ^
    • v
    Rudi, I'm not a fan of the "experiment" language they used, but it's a red herring. Being able to replicate findings is indeed very important when doing these studies. In many environments, academics must keep their original raw data on hand for x number of years in the case of challenges. It's considered a norm to share such data when requested, and if your research is in any way supposed by state or federal funds, i believe that the researcher is required to make that data available since the public technically owns it.

    In terms of your second point, this is where my brain starts to short-circuit. Who should be counted, when and how is verrry subjective. That why we have "experts" who figure these things out... My guess is that this is where we'll see the resulting huge gap in numbers and allegations of political tampering that may unfortunately be impossible to resolve objectively.
    • ^
    • v
    I don't find Burnham's defense so weak. For example, Pajamas Media criticizes the high number of reported suicide bomber victims, because such bombings are generally well reported. Maybe some of these victims were actually killed by IEDs or collateral damage from other ordinance, but with 80% of the deaths confirmed by death certificates, the death count would seem to be strong, whether or not the cause is. Discount the 655,000 number by 20% and you have 100% confirmation by death certificate. That's still 524,000 deaths, a devastating toll.

    And the challenges to the Saddam-era death rate, while interesting to contemplate, are in line with other estimates, as Burnham notes. It's a pretty young population, so not inconceivable. But the key point is that the death of a family member is a very finite endpoint. In almost all cases, the family would be hard pressed to claim a nonexistent death or to underestimate the number of still-living family members, especially with confirmation by death certificate.

    The most legitimate criticism of the study is actually offered by one of the commenters, that the study used to few clusters, possibly only 1/5 as many as needed to legitimize the figures. Fine. Let's look at a bunch more clusters, rather than sweeping the whole issue under the rug as if it's a political talking point. Maybe it is. Maybe it isn't. More research can determine that.

    However, this research was based on face-to-face interviews with 1,880 families. We regularly cite, use and trust surveys of far fewer individuals in applications from political polling to product development to entertainment ratings.
    • ^
    • v
    Ids - On the point of releasing data, Munro is wrong in that the raw data was released last February.
    The second issue is also addressed in this same article.
    http://intl.emboj.org/nature/journal/v446/n7131...
    Nature 446, 6-7 (1 March 2007) | doi:10.1038/446006a; Published online 28 February 2007

    Death toll in Iraq: survey team takes on its critics

    Jim Giles
    Top of page
    Abstract

    Raw data should settle arguments over study methods.
    ...
    The controversy creates extra interest in the authors' decision, made last week, to release the raw data behind the study. Critics and supporters will finally have access to information that may settle disputes.
    ...
    The raw data may also help address a fear that some researchers are expressing off the record: that the Iraqi interviewers might have inflated their results for political reasons. That could show up in unusual patterns within the data.
    • ^
    • v
    From the "Data bomb" article:
    Still, the authors have declined to provide the surveyors' reports and forms that might bolster confidence in their findings. Customary scientific practice holds that an experiment must be transparent -- and repeatable -- to win credence. Submitting to that scientific method, the authors would make the unvarnished data available for inspection by other researchers. Because they did not do this, citing concerns about the security of the questioners and respondents, critics have raised the most basic question about this research: Was it verifiably undertaken as described in the two Lancet articles?

    Munro claims that the data isn't made available, this is an outright LIE.
    Here are some links to prove the data was available last year.
    http://lancetiraq.blogspot.com/2007/04/data.html
    http://lancetiraq.blogspot.com/2007/04/data-now...
    http://www.jhsph.edu/refugee/research/iraq/#Rel...
    • ^