Friday, October 12, 2007

An exercise in experiment design

The Skeptic's Society has been putting up YouTube videos in which Michael Shermer (using his best cheesy documentary voice) investigates claims. These are all standard skeptical topics. Here's one where he tests remote viewing. (Below is part two, but there's also a part one that may be of interest.)

If you didn't have time to watch that, the remote viewing demonstration consists of everyone drawing stuff randomly, while trying to visualize a picture inside a sealed envelope. In the mean time, the workshop leader (who knows what's inside the envelope) goes around the room making suggestive comments. After the session is over, the leader opens the envelope, reinterprets some of the drawings so that they seem to match the picture, and quietly discards all the misses. In summary, the demonstration consists of almost every mistake in the book.

Michael Shermer got them to conduct an experiment in which the top-rated remote viewers would try to guess a picture of his choosing. The guesses are all over the place, but after the picture is revealed, the pictures are quickly reinterpreted as if they were completely right from the beginning. At this point, Shermer seems pretty exasperated.

I was kind of disappointed at this experiment. I think I could improve upon it. First, it could use some double blinding, such as keeping Michael Shermer outside as they draw. I guess I can see why they might have wanted Shermer in the room for the video... But what this experiment desperately needs is some form of control. In the remote viewing experiment, how do we know whether that remote viewing does any better than chance? The remote viewers probably are indeed better than average at drawing things that resemble a randomly chosen picture, but how would we know whether some psychic phenomenon was going on?

If I designed this experiment, I would have several separate people (we'll use non-skeptical volunteers, just to throw a bone) pick out several different pictures. A random picture would be put into the envelope. After the drawing session, the pictures are shuffled, and all given to the leader, who picks out the one picture he thinks was in the envelope. And of course, we repeat it several dozen times or until the error bars are sufficiently small enough. Compare to chance, leading to the conclusion: nothing of interest!

But of course. Even without having actually done the experiment, I already know there's nothing of interest. Scientific theory strongly suggests remote viewing better than chance is impossible. Oh, sure, empirical evidence ultimately trumps theory, but empirical evidence can be mistaken, especially when your methodology is bad. And you have to remember that theory is so strong precisely because it is supported by lots of empirical evidence.