Thursday, January 15, 2009

Strange homeopathy averaging

I was just reading eSkeptic, which is skeptic Magazine's free weekly internet newsletter. This week's article is "Homeopathy — Still Crazy After All These Years" by Harriet Hall, MD. I always find her writing to be very entertaining. Go read it on your own if you're interested, because I'm about to go on a tangent. Harriet was puzzled by the following:
One recent meta-analysis claimed to have found that homeopathy worked better than a placebo in general, but that it didn’t work better than placebo for any specific condition. I’m still trying to wrap my brain around that. That’s like saying broccoli is good for all people but it isn’t good for men or women or children. Other meta-analyses have been negative, especially the ones that looked at only the higher quality studies. A recent editorial in the British medical journal Lancet proclaimed “The end of homeopathy."
How is that even mathematically possible, you ask? I can answer that! I'm not familiar with the paper in question (I think this is it), but I am very familiar with recreational mathematics, and I have a good idea of how it might be possible. Consider the following:

Let's say we're trying to compare two medicines, which we will refer to as "homeopathy" and "scientific medicine", and their effects on two different conditions, which we'll call "the common cold" and "cancer". So what we do is we set up four different experiments, pairing each of the two medicines with each of the two conditions. Obviously, we would use double-blinded, placebo-controlled tests, but I'm going to skip those details. Let's say that we find that scientific medicine is effective 50% of the time on the common cold, and 5% of the time on cancer (these numbers are entirely fictional). And let's say homeopathy is effective 20% of the time on the common cold and 2% of the time on cancer. We might draw a table of the data like this:

But wait, the average for homeopathy is much higher! Why? It's because when we tested scientific medicine, we tested it on 100 people with the common cold, and 900 people with cancer. When we homeopathy, we tested it on 900 people with the common cold, and 100 people with cancer. Homeopathy did better than scientific medicine because scientific medicine was mostly tested on harder-to-cure conditions, like cancer, while homeopathy was mostly tested on the easy-to-cure stuff like the common cold.

Should we go by the average, or by the results for the individual conditions? Easy. We go by the results for individual conditions. The overall average can be skewed, because, apparently, homeopathy is tested much more often on the types of diseases that are more likely to cure themselves.

This might not be the answer to Harriet Hall's question, but it's a possible answer. Or perhaps she's just referring to a clearly flawed lie made by a homeopath. Frankly, either way, it doesn't speak too well of homeopathy.