The Shrinking Confidence in the Lancet Mortality Study

Michelle Malkin has brought a great deal of attention to a Harvard statistician's review of the infamous Lancet mortality data (estimating 640,000 casualties), in which he concludes:

the Lancet authors “cannot reject the null hypothesis that mortality in Iraq is unchanged.”

That's moderately damning, but I believe that beyond the weak statistical case, there is a systematic bias in the study as well - a bias that I haven't seen mentioned elsewhere. As you may recall, the study employed "cluster sampling" when it conducted its polls: pollsters went to various locations in Iraq, interviewing clusters of residences at each location. Much of the early debate centered around the validity of the cluster sampling methodology, but I don't think that was the problem here.

The real problem was in the polling itself.

The Lancet article was preceded slightly by a report submitted to MIT. In Appendix B Collecting the Data of the MIT report, we find this description of the polling:

Once in the clusters, the teams faced suspicion initially, especially at the first house selected in the random process. Lengthy explanations of the purposes of the survey—and that it would help the Iraqi people—were necessary to allay fears. In some areas, people were more welcoming, and all but a very few of the entire sample were eventually very cooperative.

This should raise the hackles of polling pundits. First, the "lengthy explanations" amounted to a sales pitch - they had to persuade the residents that it would be good for the Iraqi people. In other words, they essentially told the residents that they should answer in a way that benefited Iraqis. I leave it to you to decide what sort of answer that might be.

Second, note that the response of the first house was different than the responses of other houses in the cluster. This means that the polling of each house in the cluster was not independent. Apparently word of the survey traveled throughout the community while the first house was being interviewed, so that most households in the cluster knew that the survey team was in town looking for reports of deaths. Again, I leave it to you to guess how the typical Iraqi family might respond.

David Kane's journal article doesn't claim that the Lancet authors' conclusion was wrong - just that it wasn't as likely as they claimed it was. But if you look at the polling methodology itself, I think the evidence for systematic bias is very strong. And that does imply that their conclusion was wrong.