Document drop: A new critique of the 2004 Lancet Iraq death toll study
Update 9:30pm Eastern. Shannon Love at the Chicago Boyz blog called foul on the Lancet 2004 study early on and, with vindication, reacts to David Kane’s new analysis of the 2004 Lancet Iraq death toll study: “Kane shows that if the Falluja cluster is included in the statistical calculations, the confidence interval dips below zero, which is a big no-no. Since the study’s raw data remain a closely guarded secret, Kane cannot be absolutely certain that the inclusion of the Falluja cluster renders the study mathematically invalid…but that’s the way to bet. In science, replication is the iron test. I find it revealing that no other source or study has come close to replicating the original study. All my original points still stand. Ah, vindication is sweet.”
Via Cox & Forkum, a fitting cartoon flashback:
***
One of the most useful roles of the blogosphere is its service as an open-source intelligence-gathering medium. You can draw on the expertise of people around the world at the touch of a button. We saw this with typography experts during the Rathergate scandal; Photoshop experts during the Reutersgate debacle; and military experts during the Jesse Macbeth unmasking.
Now, it’s the statisticians and math geeks’ turn. Remember that massively-publicized 2004 Lancet Iraq death toll study? It was cited in nearly 100 scholarly journals and reported by news outlets around the world. “100,000 Civilian Deaths Estimated in Iraq” blared the Washington Post in a typical headline.
There were attempts made by lay journalists to debunk the 2004 study (as well as the 2006 follow-up study that purported to back up the first). But none of those dissections comes close to a damning new statistical analysis of the 2004 study authored by David Kane, Institute Fellow at the Institute for Quantitative Social Science at Harvard University. I read of Kane’s new paper at this science blog and e-mailed him for permission to reprint his analysis in its entirety here so that a wider blog readership could have a look. He has given me his permission and adds that he welcomes comments and feedback. He’ll be presenting the paper at the Joint Statistical Meetings in Salt Lake City on Monday — the largest conference of statisticians in North America.
Much of the math here is mind-numbingly complicated, but Kane’s bottom line is simple: the Lancet authors “cannot reject the null hypothesis that mortality in Iraq is unchanged.” Translation: according to Kane, the confidence interval for the Lancet authors’ main finding is wrong. Had the authors calculated the confidence interval correctly, Kane asserts that they would have failed to identify a statistically significant increase in risk of death in Iraq, let alone the widely-reported 98,000 excess civilian deaths.
An interesting side note: as Kane observes in his paper, the Lancet authors “refuse to provide anyone with the underlying data (or even a precise description of the actual methodology).” The researchers did release some high-level summary data in highly aggregated form (see here), but they released neither the detailed interviewee-level data nor the programming code that would be necessary to replicate their results.
Kane has sent his paper to Lancet. But the blogosphere need not wait for Lancet to complete its review. If you’ve got a statistics background or know someone who does, have a look. Kane’s e-mail address is dkane-at-iq.harvard.edu. He’s a blogger himself at EphBlog.
***
See what others have said
Note from Michelle: This section is for comments from michellemalkin.com's community of registered readers. Please don't assume that I agree with or endorse any particular comment just because I let it stand. A reminder: Anyone who fails to comply with my terms of use may lose his or her posting privilege.
Trackbacks
- empty rhetoric body count, redux «
- Lump on a Blog » Blog Archive » Bad Math, Bad Science: The Impact on the Lancet Iraq Death Toll Study
- New Critique of the Lancet's Civilian Death Claims : Weblogs
- Bill's Bites
- The Crossed Pond » Lancet Iraq death toll study full of holes?
- Chicago Boyz » Blog Archive » Vindication Is So Sweet
- Right Wing Nut House » SCIENTIFIC DEBUNKING OF LANCET STUDY: DOES IT REALLY MATTER?
- “100,000 Civilian Deaths in Iraq” at Conservative Times--Republican GOP news source.
- EphBlog
- “Shock troops” - Neptunus Lex - The unbearable lightness of Lex. Enjoy.
- Right Voices » Blog Archive » Document drop: A new critique of the 2004 Lancet Iraq death toll study.
- The Conjecturer » News Brief, Work Is Crazy So This Is It for a Week Edition
- Magic Statistics - “I accept no responsibility for statistics, which are a form of magic beyond my comprehension.” — Robertson Davies » Was 2004 Lancet study correct to toss out Falluja data?
- Survey says…! « Volunteer Opinion Journal
- JunkYardBlog
- Pajamas Media
- Publius Pundit
- Ft. Hard Knox » links for 2007-07-26
- links for 2007-07-26 | MY Vast Right Wing Conspiracy
- Crooked Timber » » Alice in Wonderland and the Lancet study
- Random Jottings
- Club Troppo » Missing Link - Saturday 28 July 2007
- NeoCon News » Global War on Terror Weekend Recap
- Trusted.MD Network
- New Critique of the Lancet's Civilian Death Claims · Articles
- New Critique of the Lancet’s Civilian Death Claims · New York Articles
- Mudville Gazette
- QT Monster's Place
- Lancet study validity in doubt - Page 16 - Debate Politics Forums
- Lancet study validity in doubt - Page 17 - Debate Politics Forums
- Is waterboarding torture? - Page 72 - Debate Politics Forums
- Lancet study validity in doubt - Page 23 - Debate Politics Forums
- COACHEP » “Laaa, laaaa, laaaaa. I can’t hear you!”
Comments
You must be logged in to post a comment.
Categories: Document drop, Iraq
Mudville Gazette
» The five-year plan
Legal Insurrection
» Is "Finish the Job" the New "Peace With Honor"?
Sister Toldjah
» WaPo and NYT to Obama: We love ya, you stylish deep thinker
Pundit & Pundette
» Hiding the decline: Deniers made them do it
Power Line
» More From the East Anglia Archives
Founding Bloggers
» Did The New York Times Just Tell Glenn Reynolds To “STFU”?

Riehl World View
» Google Refuses To Remove Michelle Obama Image
























Wow, all that math made my brain hurt……….way too complex.
Raging Republican… I think that this is one point on which we can agree.
That is a lot of math.
MM – thanks for the report…but geez…my head hurt just from reading your DESCRIPTION of the findings…. LOL
That’s why I deal with people – not numbers.
The basic argument here is whether or not Fallujah should be included in the cluster sample, since it’s inclusion/exclusion will affect whether or not we can reject the null hypothesis or not.
All the stats that follow are just the results of what happens when Fallujah is included, so you can pretty much just ignore them, and count on the fact that they’re right if article is going to be published.
It’s an intersting and apt critque, but I’m not so sure this actually debunks the Burnham study as Michelle seems to suggest.
What I think is most damnning in the article is the statement that Burnham and Roberts (whom I have as professors at JHU) refuse to release the data. I recall a lecture earlier this year in which Roberts said that accurate data collection may help say thousands of lives (this was in reference to the Rwandan refugee camps) so it seems odd to me that he and Burnham would withhold data like this. Given Roberts’ past politcal actions (and he was never shy about talking about them in class), it does seem as if they’re hiding something.
This is extremely disapointing, as both professorts have done very good work in the past, that has help thousands of refugees in the past, and I think this will (at least it should, but may not be given the liberalism that has infected academia) forever be a blackspot on their careers.
I am like most of you.
My head is still spinning form all the data.
Math/Statistics were never my forte.
I think the greater damnation of these numbers and studies is the sudden need to care about how many Iraqi’s are dying.
No one on the left raised so much noise before the war, when it was Saddam’s regime who was doing the damage. But now, since the war started, they feel the need to count the dead and attribute it to Bush or the US military – when most ( and I’m hypothesizing here ) of the casualties of civilians are caused by insurgents & terrorists.
But it’s always the US who gets blamed.
Yes, some may argue that if we weren’t there in the first place, these deaths wouldn’t have occurred. But that dismisses the atrocities committed by the previous regime.
I think you also have to think about whether or not they omitted Fallujah on purpose because it would change their results. If they thought that Fallujah was an outlier that would bias the data, then why bother collecting it their? It seems as if the omission of the Fallujah data occured after they had collected the data and did some preliminary analysis. This is very bad statistical methodology, which they likely wouldn’t do unless they had some ulterior motive.
…and another thing. I think the prewar mortality rate that Burnham used was suspicious. Yes, he probably got it from a reputible soruce, but I think you have to question these numbers when you’re dealing with a totalitarian governement. When opposition figures just suddenly disapear and are later found in a mass grave, are these numbers counted. My guess is no, since most of these people would have just vanished without any record of their death.
taylork said:
I think it’s called “an agenda” … LOL
I’ll read it at work (where I edit such papers) but kudos to Dr. Kane!
Debunking anti-Americanism in the academy? He must be the loneliest guy at Harvard.
Just quickly scrolling past all that stuff made my brain hurt.
I would like to see a companion qualitative study completed by qualified experts. A suggested title:
“Fighting Strategies of Al-Qaeda In Iraq: An Analysis of Defensive and Offensive Technique of Enemy Combatants In Urban and Rural Settings”.
Perhaps this study would shed light on how and why so many civilians died. Oh but no way. We can’t *know* that because it wouldn’t reflect well on *ALL* of Islam.
These Lefty Rats who use the childish logic “If the US had not invaded, these people would still be alive” makes me sick. Fine. If you want to explain *half* of reality, I guess they have a point. I guess when they have a boil or an infection they don’t lance or medicate because they don’t want the side-effects. “oh! That awful medicine gave me cotton mouth. If i hadn’t taken it I wouldnt have cotton mouth!”
Pish.
That’s the essence of the issue.
The problem with a lot of liberals is that they’re mentally lazy. A compounding problem is that they lazily tend to assume that everyone else is as mentally lazy as they are.
The “98,000 casualties” concoction in this case is an example of the above. Some people began with an ideologically motivated, cherished preconclusion (the USA must be somehow responsible for scores of thousands of innocent Iraqi deaths), then worked backwards with some fun-and-games-with-numbers to “prove” it.
It would’ve been too much mental effort for the authors of the original “study” to have phrased the base proposition in the form of a question, and then work forward. Besides, the answer might not have suited them.
What is more, the complexity of Kane’s analysis demonstrates that the authors of the Lancet “study” likely knew that few people in general, and certainly nobody inclined to share their beliefs, would undertake the effort needed to test their findings.
That they played “hide the ball” with their data and methodology was just a dishonest insurance policy against the possibility that somebody like Kane just might call them out.
There is no justifiable reason for any honest scientist or researcher to refuse to disclose that upon which he or she bases his or her conclusions. The only explanations for such a refusal are that the authors either know that the data doesn’t back them up, or that they lack confidence in their conclusions but still want to present them as conclusive.
Both are variations of intellectual dishonesty; the former intentional, the latter reckless. It’s no different from Mike Nifong refusing to disclose exculpatory evidence from the defense attorneys in the phony Duke “rape” case: the facts might get in the way of the narrative.
It was printed in scholarly journals, and they refuse to release the underlying data? I always thought it was REQUIRED when you did that…
Without the underlying data, the whole Lancett study cannot be given any credence.
How they decided which areas to include in the study, and which to leave out, and where they got the death data from, is needed to understand whether this is a good analysis or not.
If you had chosen pre war death areas to include those GASSED by Saddam… I’m sure the numbers would have changed dramaticly… they are attempting to extrapolate numbers from a data set which is not large enough.
The left did the same thing in Vietnam. Why shoud anyone be suprised,they hate thier country. Anything to make America look bad is thier motto.
This, above all, is the real problem. Of course people can lie to you if you don’t know what they are doing with the numbers! Somehow it’s shameful to admit that you can’t read, but perfectly okay to admit that you can’t do math. And people who do know math are necessarily “geeks” – because oh-it’s-so-complicated — NOT.
Mathematics is an essential component of human knowledge. Knowing math (and we aren’t even talking complex math — we are talking basic numeracy here) is as important as knowing how to walk, read and write. If you “can’t do math”, then you never acquired the basics of logical thinking. It’s that simple.
And because of this widespread innumeracy (that’s what mathematical illiteracy is called, there’s a good book written about this: http://en.wikipedia.org/wiki/Innumeracy_%28book%29), there are all kinds of possibilities for people to pull a fast one on you — both from the right and from the left, or not even from a political standpoint — that’s how casinos make their money, after all !
If you don’t have a good grasp of mathematics — don’t admit to it. It IS shameful. Go and learn it. Then you’d be able to read papers like this and Judge. For. Yourself.
Gee. And I always thought taking a simple body count through the rubble was the most accurate assestment possible. Silly me.
James Greenidge
Queens NY
Statistics – like war – is by nature messy business. All statistical analyses come with the built in caveat that the results you’re looking at may or may not be accurate. That’s where the confidence interval comes into play.
This does not necessarily refute the original study, but rather casts doubt on their claims. What I find most telling here is that the authors refuse to release their data. I don’t like it when Bush does it, and I don’t like it when these guys do it, either.
One point worth making here is the importance of being consistent with methodology. How was the CMR before invasion estimated? And after? I’m too lazy to read the original paper, but I suspect they were counted by completely different means.
One other point to make – The Lancet is a medical journal, not an epidemiologic journal. This may not seem like an important distinction, but it is: the original study probably wouldn’t have cut the peer-reviewed mustard if statisticians and epidemiologists were doing the reviewing rather than doctors. Doctors may be smart, but they don’t know statistics.
My theory is that the truth is somewhere in between these two different analyses. The violence of war paired with a collapse in the healthcare system would no doubt lead to higher death rates. But probably not as high as the original study projected.
This study was short on actual statistical analysis and long on unsupported conclusions, which is probably why the study was only published Online and not in the actual Journal itself. (However, I seriously doubt that this kind of analysis would have passed muster by my old Statistics teacher. Each conclusion should have a piece of statistical analysis to back up that conclusion.)
Michelle also includes the text of Dr. Kane’s analysis, and it confirms some of the problems with the Original lancet study. I didn’t get off the first page before I found three major problems.
The first major problem is in the fourth paragraph under the “Entire Findings” section, citing the Lancet findings, the fourth sentence states “We estimate that 98,000 more deaths than expected (8,000 – 194,000) happened after the invasion outside of Fallujah and far more if the other Fallujah cluster is included.” in layman’s terms, they are saying that as many as 194,00 people died as civilian casualties after the invasion by Allied Forces, and very likely more since the casualties from Fallujah (one of the most violent ares of Iraq) are not included in this estimate. And as Dr. Kane notes in the fifth paragraph, second sentence, “The authors did not provide a confidence interval for excess mortality which included the data from Fallujah and have declined my requests to do so.”
Failing to provide raw data for a subsequent analysis isn’t just unusual — it’s just not done. Researchers routinely provide the raw data for later researchers — to refuse to provide data is usually a big red flag that the researcher is trying to hide something. I had one incident where I wrote an e-mail in 1998 to a researcher asking about a study he had done and received over 50 pages of raw data attached to the return e-mail — THAT’s how routine this is. Refusing to release their raw data makes this study highly suspect.
So, ultimately, it’s extremely difficult to figure out how exactly these authors came up with such a high confidence level (95%) when they have (a) deliberately left out the Fallujah data, and (b) won’t release their raw data — not to mention extremely suspicious.
The second major problem is in the fourth paragraph under the “Entire Findings” section, in the seventh sentence which states “Most individuals reportedly killed by coalition forces were women and children.” While a statistical analysis could infer that a certain population group incurred the largest casualties (such as “women” or “children” or “men”), to group the top two population groups together (i.e., “women and children”) is also not a standard statistical procedure and highly suspect. This sounds more like either the interview subjects feeding information to the researchers, or the researchers drawing unsupported conclusions. (And, again, since they will not release their raw data for review, these conclusions cannot be supported by data.)
The third major problem is just with the sheer numbers involved. The Lancet study believes that probably 98,000 civilians — and perhaps as many as 194,00 — died post-invasion “and far more if the other Fallujah cluster is included.” A quick web search shows that the city of Fallujah has a current population of 284,500 people. But does “far more” casualties mean 50% more — 100% more? suppose it means only 50% more — that would mean between 49,000 and 97,000 people died in Fallujah post-invasion; between 17.2% to 34% of the population of Fallujah died — and the media failed to notice?
Where are all these bodies? Where are the mass graves and burn pits that Allied Soldiers would have to run to dispose of all these casualties? Plus, if you add my “guesstimates” of Fallujah estimates to their guesses of casualties you get from between 147,000 to 291,00 civilian casualties. If we had that percentage of the population “disappear” we’d be talking about the populations of Chicago, or Los Angeles, or even Manhattan disappearing. I think there might be more than a little talk about it in the Press — it would be hard to hide that percentage of the population disappearing.
Even if you accept the numbers at face value, they just don’t add up. And if you try to run their numbers yourself you can’t, because they won’t release the data to you — something only a person who had something to hid would do.
As a final note, two of the Lancet’s studies authors are from Al-Mustansiriya University, Baghdad, Iraq (R Lafta MD, and J Khudhairi MD). The two Iraqi physicians (Lafta & Khudhairi) were, as the Lancet study notes, “R Lafta was involved in study design, hired, trained, and oversaw the interview staff, led one of the two study teams, coordinated all logistical aspects of the study, and had a central role in data interpretation and preparation of this report. . . J Khudhairi was involved in the study design,interviewer training, and oversaw one of the two survey teams in the field.”
Of the five contributors to this report, only the two Iraqi physicians had input to the actual interview process; the other three authors interpreted the data provided by the two Iraqi physicians.
Funny, I seem to remember something about Middle Eastern physicians and radicalism recently. I also seem to remember something about Iraqi stringers feeding fraudulent information to media journalists — surely that would never happen to academics?
Yeah, right. What’s that phrase — “You’ve been Pwned”?
Mark Twain said it best:
“There are lies, damn lies, and then there are statistics.”
and Homer Simpson said it almost as well. “Oh, people can come up with statistics to prove anything. 14% of people know that.”
Numbers do not lie.The commie-puke/al-Qeada loving slugs do.
jrlingreenbay made an interesting point about “the left” not caring about how many people Saddam killed. I’ve been scouring the mass grave sites but have yet to see anyone compile a number. If anyone could provide a link to a site with data, I’d appreciate it.
As for the Lancet study, everyone needs to keep in mind that they were referring to “excess mortality.” That means those deaths not directly caused by coalition forces. The implication being that infrastructure has been destroyed, farms uprooted, cattle dead, etc. and that the lack of basic services caused more people to die than would have if these things were working.
I didn’t see anything in the study author’s paper wherein he justified why a “zero null hypothesis” was invalid.
Prior to the invasion, the human rights groups were saying that the UN sanctions against Iraq were killing millions. The sanctions being lifted was a direct result of the invasion. So aren’t we actually saving millions of lives?
I got Bs in two difficult stat classes, but my brain goes all topsy-turvy glancing over these studies. Yay talking points I’ll have to research!
To fdr_jr who likes to insult those of us with intelligence.
I wager to bet you cannot play any concerto on a concert grand piano.
Guess that makes you smarter than me huh?
Oh, and let me add, speaking for others.
We can dang sure count our dividends, earned income from Trust accounts, know our rates from our money market/mutual funds, shall I continue?
It’s people like fdr_jr that have to insult others to make themselves feel superior.
Nice try though.
Since we have a real, scholarly, technical paper on hand, I’d like to call your attention to the last sentence in the Abstract, on the first page. It says “Comments and corrections are welcome.”
I’m from the a technical field and I read such publications all the time, and it’s very common for authors to welcome debate.
So please take this lesson home with you tonight: a real scientist doesn’t ever say “there is a consensus, and the debate is over.”
What Michelle doesn’t mention is that the study came out 5 days before the 2004 election.
The Lancet denied politics was involved in the decision to publish it then – even though it came out on a Friday while the normal date for publication would have been the following Tuesday.
Here’s the NY Times article that is surprisingly balanced about the political aspects.
Oh there must be an easier way, what about a census for the living, has the new Iraq Govt endorsed or suggested a census be taken? I don’t know what Saddam and his Socialist Govt., used to determine the size of the Iraq Population, and demographics before the war for comparison now.
How would you count a sucide bomber is that a war casualty?
fdr_jr…just because everyone doesn’t have knowledge of advanced statistics does not mean they are innumerate. The statistics in this paper require several years worth of advanced training, and it’s ridiculous to think that everyone is going to be able to take the time to learn it. Being innumerate refers to not knowing basic math, not post-graduate level statistics, so get off your high horse.
Noticed how he did a fast split?
Guess he couldn’t handle the heat.
Data analysis can be used to support trends, or trend analysis can be used to support data. The problem is that many people use analysis methods to prove, or disprove their theory. Like political polling, depending on the question asked, how the question is asked, the people you ask, and what the response options are, you can make the data support any answer you wish. You can also exclude what some would call ‘eronious’ data. Political polls do it all the time.
In an analysis such as this, you attempt to identify a trend with known numbers (questionablei this case), then apply the ‘assumption’ that it is ‘constant’ accross the board. Using good data, you get acceptable results. Using assumed data you can get garbage.
Example, the captian of the Exxon Valdez was drunk. Therefore, all ship’s captains drink, or are drunk. This is using a broad stroke and the result is almost always wrong.
This is where the taking of one exceptional action with high casuaties and adding it to a few other cities with casualties, then applying the trend broadly accross the entire population, many areas where no action took place, exagerates the final assumption.
Example, take the murder rate of the top ten cities in the U.S. for the year 2001. Include the 3,000 victums of the twin towers for NYC. Come up with an average murder rate per 100,000 people for the top ten cities. Then apply the same percentage for murder per 100,000 people accross the entire nation. You will find the murder rate for the nation increased by roughly 50% to 60%for the year 2001. But, it is entirely a false assumption based on the fact you used a rare event (an oddity)to support your conclusion.
I hope this made sense, it has been a long day
As for the assumption the U. S. caused all the civilian casuaties. Does anyone remember all the anti-aircraft fire, ground missles, small arms, etc.. fired at our aircraft. That stuff had to come down somewhere. It didn’t hit the planes:-)
Just one add on, sort of related, part of the group that approached the U.N. with man made global warming did not supply data with their conclusions. A few of them stated “we will make up the data to support our conclusions and offer it to the U.N at a later date”. Now, maybe something was lost in the translation, but…
Wile C:
I’m afraid you’re mistaken.
“Excess deaths” refers to people who died because of the invasion.
In other words, they speculate that these people would have been alive if we hadn’t invaded.
In fact, that study says that most of the deaths in Iraq since the invasion have been violent deaths and laughably, 80% of those were caused by airstrikes!
You are correct that they added on to the violent deaths those caused by poor medical care, bad nutrition, etc.
Thanks for this information Michelle. We all figured the Lancet Iraq Death toll study was bogus. It’s nice to have a scholarly paper to help validate our doubts.
Roberts and his gang and Ward Churchill have disgraced academia. They should be shunned.
The only logical reason why the study authors would absolutely refuse access to the raw data is that said data, and the subsequent study conslusions, cannot stand expert scrutiny. That alone is sufficient to dismiss the findings. (After all, the data do not represent, say, the outcome of scientific experimentation that could have future business and financial implications.)
There’s a typo in the link to the group blog that David Kane writes for has a typo. It should be “http://www.ephblog.com/“.
The guy who writes the deltoid blog is a sad little moonbat who rarely writes anything not intended to deceive.
I’m a mathematician, but my observation here is that you can in fact be innumerate and still be 99.4% certain that the Lancet study was bogus, simply from the fact that…the Lancet authors “refuse to provide anyone with the underlying data (or even a precise description of the actual methodology).”
NO serious scientist or mathematician would submit a study with such an unacceptable condition. Nor would they take such ANY study seriously if submitted by any of their peers.
It has been 5 years since my MBA Statistics courses, so I am a tad on the rusty side, but credible studies eliminate the outliers. Is Fallujah an outlier? If it skews the data, as suggested, then it is an outlier.
Also, scientists/mathematicians are like everybody else, in that they have egos, and prefer more fame to less. If they are right, they would shout it to the world, and welcome scrutiny.
In this case, the null hypothesis is that they have null to hide. The alternative hypothesis is that they are hiding something, and that inviting scrutiny might cause their analysis to crack.
eric
http://www.blacktygrrrr.wordpress.com
Lancet is no longer a scientific periodical. It is the Vanity Fair of medically inclined Moonbats. It has no value whatsoever in real science.
I reckon’ it all comes down to being a statistical study, which neither the leftards nor the MSM understand. Remember, the original study said it was anywhere between 8K and 198K civilians killed, so the Moonbats decided to take the mid-point and call it 100K, just to assail Bush, our troops, and Operation Iraqi Freedom.
They had no clue how the study was conducted, nor did they want to know.
No point rehashing the entire idiocy of what know regarding the gathering of data, but, another thing the MSM doesn’t want to know, is that study, using actual data, rather then statistical, by a Jordanian doctor that showed 50,000 children were dying each year during sanctions, because of Saddam’s regime.