Sunday, April 15, 2007
Data!
The authors have made a version of the data for Lancet II available to me. Thanks! To be honest, I had worried that they wouldn't do so. I was wrong to worry. I appreciate the authors sharing this data with me (and others). I also thank them (in advance!) for answering the questions that I and others will have.

A central value of any good scientific community is transparency of methodology. If you want to claim that X is true, you need to explain in detail why X is true and demonstrate how other scientists, especially those who disagree with you, can replicate your results. The Lancet authors have taken the first step along this road. Good for them.

posted by David Kane at 12:29 PM

I think you're seeing what you want to see here Rudi. You don't seem to be paying attention. The article even says:

“Under pressure from critics, the authors did release a disk of the surveyors' collated data, including tables showing how often the survey teams said they requested to see, and saw, the death certificates. But those tables are suspicious, in part, because they show data-heaping, critics said.”

That disk is the data you're talking about. Certainly the NJ article is not hiding that this data has been released. There's even a sidebar illustrating a piece of that released data (which btw, seems to be suggestive of fraud):
http://nationaljournal.com/njcover.htm#

The passage you quote is clearly referring to other aspects of the data and forms which have been kept secret, not the same as the data disk that was released. You gloss over the particulars here with a generalized “the data” in order to call Munro a liar, as if “the data” covers everything in question. It doesn't.

And if you want to go there, it's really Burnham who, if anyone, should be called on outright lies for his claims in the Pajamas Media piece. He claims for example that with the 2004 and 2006 studies:
“The methodology was the same, though we increased the sample size and the number of clusters [from 33 to 47].”

So supposedly they were exactly the same, but the 2006 one was just larger (and therefore supposedly better). Actually, the methodology was quite different in 2006 (and more suspect). The pages on the link that Rudi posts above to rhul actually have many pages on this and discuss serious problems with the 2006 methodology that don't directly apply to the 2004 methodology: because they were different. The 2004 one says it used GPS units to select each house, not his “Main Street” thing that the 2006 one did.

Then Burnham even claims that “The 2006 data from January 2002 through Sept 2004 (end of the 2004 study) were virtually identical to the 2004 studies.”

Apparently this should allay concerns: The methodologies were the same and the results were “virtually identical” In fact, the same website debunks this claim (outright LIE?) in detail here:
http://www.rhul.ac.uk/economics/Research/confli…

The two studies actually show very different things for the same period. You can even see this in the NJ article graph on the right side. The 2006 one looks way out of whack.

I will look into the data analysis, here is another web site by academics involved in the spat.
http://www.rhul.ac.uk/economics/Research/confli…

Still, the authors have declined to provide the surveyors' reports and forms that might bolster confidence in their findings. Customary scientific practice holds that an experiment must be transparent — and repeatable — to win credence. Submitting to that scientific method, the authors would make the unvarnished data available for inspection by other researchers. Because they did not do this, citing concerns about the security of the questioners and respondents, critics have raised the most basic question about this research: Was it verifiably undertaken as described in the two Lancet articles?

Munro claims that the data isn't made available, this is an outright LIE.
Here are some links to prove the data was available last year.
http://lancetiraq.blogspot.com/2007/04/data.html
http://lancetiraq.blogspot.com/2007/04/data-now…
http://www.jhsph.edu/refugee/research/iraq/#Rel…

Nature 446, 6-7 (1 March 2007) | doi:10.1038/446006a; Published online 28 February 2007

Death toll in Iraq: survey team takes on its critics

Jim Giles
Top of page
Abstract

Raw data should settle arguments over study methods.
…
The controversy creates extra interest in the authors' decision, made last week, to release the raw data behind the study. Critics and supporters will finally have access to information that may settle disputes.
…
The raw data may also help address a fear that some researchers are expressing off the record: that the Iraqi interviewers might have inflated their results for political reasons. That could show up in unusual patterns within the data.

And the challenges to the Saddam-era death rate, while interesting to contemplate, are in line with other estimates, as Burnham notes. It's a pretty young population, so not inconceivable. But the key point is that the death of a family member is a very finite endpoint. In almost all cases, the family would be hard pressed to claim a nonexistent death or to underestimate the number of still-living family members, especially with confirmation by death certificate.

The most legitimate criticism of the study is actually offered by one of the commenters, that the study used to few clusters, possibly only 1/5 as many as needed to legitimize the figures. Fine. Let's look at a bunch more clusters, rather than sweeping the whole issue under the rug as if it's a political talking point. Maybe it is. Maybe it isn't. More research can determine that.

However, this research was based on face-to-face interviews with 1,880 families. We regularly cite, use and trust surveys of far fewer individuals in applications from political polling to product development to entertainment ratings.

In terms of your second point, this is where my brain starts to short-circuit. Who should be counted, when and how is verrry subjective. That why we have “experts” who figure these things out… My guess is that this is where we'll see the resulting huge gap in numbers and allegations of political tampering that may unfortunately be impossible to resolve objectively.

Criticism like this happens all the time in academia and is usually ignored by a general public who would be bored to tears by such discourse. And amongst those who aren't playing the publish-or-perish game, you'll see attacking the very methodologies accepted by the academic community. Like with the topic of global warming, in this particular case, it's just playing out more publicly because of topic.

Let's see if the challenge stands up.

(1)Still, the authors have declined to provide the surveyors' reports and forms that might bolster confidence in their findings. Customary scientific practice holds that an experiment must be transparent — and repeatable — to win credence. Submitting to that scientific method, the authors would make the unvarnished data available for inspection by other researchers. Because they did not do this, citing concerns about the security of the questioners and respondents, critics have raised the most basic question about this research: Was it verifiably undertaken as described in the two Lancet articles?
…

(2)The authors should not have included the July data in their report because the survey was scheduled to end on June 30, according to Debarati Guha-Sapir, director of the World Health Organization's Collaborating Center for Research on the Epidemiology of Disasters at the University of Louvain in Belgium. Because of the study's methodology, those 24 deaths ultimately added 48,000 to the national death toll and tripled the authors' estimate for total car bomb deaths to 76,000. That figure is 15 times the 5,046 car bomb killings that Iraq Body Count recorded up to August 2006.

The first criticism(1) is just plain absurd, a survey is not an experiment. The scientific method really doesn't apply to studies. Question the methods and statistics, but WTF does an experiment have to do with the Lancet study. The study authors didn't claim that “cold fusion” is possible.

The second (2) criticism is plausible but very weak. Discount a study because it was running behind schedule in war torn Iraq. The “main street bias” argument is plausible, but a time delay/schedule is stupid.

One other point, Munro uses the IBC numbers as a sly way to discount the Lancet study. But, IBC isn't scientific and Iraq's Interior ministry came out with a figure of 150.000 dead. A four to one discrepancy isn't as bad as 10 to one. And bringing up Soros is a red herring…

For what it may be worth, I don't typically quote from the NJ, but because I had gone on record more or less buying into the study, I believed that it was important to write about the NJ story.

Fraud, however, is another matter… Let's see how this plays out over the next couple days/weeks as this heats up.

One example:(NJ article)

According to a data table reviewed by Spagat and Kane, the team recorded the violent deaths as taking place in early July and did not explain why they failed to see death certificates for any of the 24 victims. The surveyors did remember, however, to ask for the death certificate of the one person who had died peacefully in that cluster.

A response from Kane, at his blog related to the Lancet study:

Lancetiraq.blogspot.com/
Neil Munro's National Journal article is out. I haven't had a chance to read it closely, but my quotes are not as contextualized as I would like them to be.

Kane is saying that Munro is taking him out of context.