Crusader War College

Link: http://michellemalkin.com/2007/07/25/document-drop-a-new-critique-of-the-2004-lancet-iraq-death-toll-study/

One of the methods used by Leftists to gain and keep power and influence is to exploit webs of implicit trust. This post shows an example of this.

There was a study published by the British Medical Journal Lancet that stated that there were 100,000 more deaths in Iraq than normal as a consequence of the Second Gulf War. The paper has been critiqued before, but the most devastating one is reproduced at Michelle Malkin's blog at the link above. In it, David Kane uses statistical analysis on the data cited in the original paper to show that the "confidence interval" was incorrect. The confidence interval is a measure of how different the given value is from the "norm", and thus a measure of how significant the difference is.

While I am no statistics wizard, I perused the paper at the link above, and noted that Kane correctly uses a Bayesian analysis, because nobody really knew ahead of time what the baseline risk of dying in Iraq was before the war. After some analysis, Kane notes that the confidence interval that was published is obtainable if the authors left out the statistics they gathered in Fallujah. However, because the authors of the Lancet paper have not released the raw data or the detailed analysis methodology, one cannot really review their methodology or double check the data or the results.

That bolded part is all that you really need to know about the validity of the study. Forget about the complexity of the math: One of the core requirements of science is replication and verification, so a refusal to reveal the data or the methodology after the paper is published, so that it can be double checked, is a gross violation of those requirements. While there are people who do not think that this should be enough to disqualify the paper or its authors, and reject as unproven its conclusions, that merely shows that they are ignorant of how real science works.

...

One has to realize that there is no auditing function in science, and a close look at how "the sausage is made" reveals a web of trust relationships that are vunerable to compromise. Firstly, one must realize that the scientific community is not monolithic, with every scientist knowledgeable enough about everything to be able to properly judge good or bad science. The community self-groups into smaller groups, based on the field of study, with even smaller groups subdividing into "factions" that usually form around basic differences within the field. When it comes to other sub-fields, the majority of scientists implicitly trust the findings and opinions of scientists who have decided to specialize in those sub-fields. This is a mutual trust relationship: all the scientists NOT in the sub-field will equally trust and agree to the findings and opinions of the scientists that ARE in the sub-field. Everyone winds up trusting everyone else NOT in their sub-field, with the understanding that it is no-holds-barred with regard to work within the sub-field of specialization.

This situation of there being a web of trust within the scientific community merely mirrors the web of trust that the scientific community has with other human communities within the larger society. If you read the comments on the blogs carrying this result, you'll see that people cannot fathom the math involved, and will accept the results based on implicit trust, thus creating a web of trust. What you have to realize is that, within the scientific community, scientists are also "in the dark" when it comes to judging the output of scientists not working in the sub-field that they themselves are working on.

So the question is, if the trust is implicit, is it merited? The Scientific community has come up with mechanisms that they hope will allow people to believe them when they answer "yes!"

One mechanism is the peer review, where a research paper produced by one scientist is examined and reviewed by other scientists. While the Lancet paper was peer reviewed, obviously the quality of the review is only as good as the "peer". However, there is no way to verify how good the peer is, because such information is kept secret by all scientific journals. This is supposedly to prevent the authors from getting revenge on the reviewer for a bad review. Why is that? Because they KNOW that, even in science, there is politics: people trying to get their way, and sometimes getting their way means producing bad science. However, it is obvious that the peer must come from within the sub-field simply because one doesn't want a paper needlessly rejected because the reviewer doesn't understand the field enough to tell whether it is good enough. Indeed, the whole peer review process for journals is in place because journal editors don't understand all the sub-fields enough to tell whether papers are spot-on or off-base.

Given the above, it should be clear that the peer review system is itself part of a web of trust, in that the journals are assuming that the peers will judge the paper on its merits. However, it should be obvious that all is not sweetness and light ALL AROUND: the journals do not trust the authors to not retaliate against peer reviewers. If that is so, then why should the journals implicitly trust that peer reviewers won't torpedo papers that contradict their own beliefs or their own positions? In short, when we examine the web of trust when it comes to peer reviews, we see a thread going one way, but a broken one going in the other direction. The Peer Review process is a visible mechanism that appears to correct authors of scientific papers, but there is no mechanism whatever to correct peer reviewers, and thus no way of confirming that the peer review process is being abused by peer reviewers or the journals. How can one confirm non-abuse if the essential facts (who reviewed what, the review, and who assigned what reviewer to what paper) are hidden?

How is this handled in other, non-scientific fields? In business, the truth claims of the books and the annual reports of businesses are not taken in trust: con men can make the same claims, in order to buy time to extract more funds from the marks. The strands of trust comprising the web of trust in business does not adhere to the business, but to the auditing firm that the business retains to audit their books and financial reports. The auditing firm is trusted because establishing and maintaining the web of trust attached to it is central to it's survival. This is exemplified when the auditing firm of Arthur Andersen essentially exited the auditing business when it sought to extend its web of trust to Enron and Worldcom, when those businesses were not worthy of the trust being transferred to them. Even though, per the Wikipedia entry, Arthur Andersen was eventually exonerated in court, its survival depended on an unexploitable web of trust adhering to it. When the appearance arose that the web was being exploited, everyone cut the strands on their side, and Arthur Andersen dropped into the abyss...

There is no official auditing function in Science. Nobody is certified to go over the list of peer reviewers, whose papers they reviewed, and which editors decided who would review which papers, and issue a statement that the peer review process is working. All that the journals can guarantee is that they are going through a peer review process, but cannot prove that the process is impartial and not being exploited. What would happen if it was discovered that journal editors were sending ALL articles regarding global warming ONLY to reviewers that support global warming, AND that those reviewers consistently rejected papers that showed that global warming did not exist, or which disproved previous papers that DID support global warming? The reputation of the editors, reviewers, and the journal would be seriously tarnished. The fact that such audits are vigorously opposed and not being done should be regarded as the scientific equivalent of a business refusing to have its books audited. The only reason why people react differently when Science is involved is because they have not been told that science done this way is as suspicious and bogus as a business that rejects audits. In short, they have been told that businesses that act like this are bogus, but they have not been told that science done this way is equally suspect.

Now, someone may say, "Well, one way of feedback is for scientific results to be checked independently." That is correct, and Cold Fusion is a perfect example of how the independent confirmation process worked to disprove it. However, the original researchers did publish the results, the experimental set up, and the analysis methods: Doing so was necessary to enable the results to be confirmed. In addition, the contrary results were published. The process showed that the web of trust in that subfield of science is intact and worthy of trust. However, that does not justify blindly extending the web of trust to other subfields, or to science in general. That's like saying, "Company A has a great repair department, so you can trust the repair department of Company Q." In business, we'd say "Huh???", so where is the skepticism when the same claim is made in science?

And it is in the refusal of the authors of the original Lancet paper to distribute the raw data and the methodology that proves to me that the paper is unworthy of being believed. It is the scientific equivalent of a business saying, "I was audited last year, so I don't need to this year!" Don't swallow shit because it was put into a can and labelled as chili!

Ronald Reagan advised "Trust, but Verify." The obvious corollary is, "If they won't let you verify, don't trust them." That applies everywhere, including science.

Hattip Little Green Footballs