Thursday, June 3, 2010

Experts in skepticism

Several months ago, I proposed that a court of law is another "way of knowing" just like science.  I was going to start a series of posts comparing law to science and skepticism, but I got distracted, as usual.  But I never forget a topic idea that I don't want to forget (ie I write them down).

There are two great things about law as a method to determine truth.  The first is that law is practical, not philosophical.  A court of law never has to worry about silly questions like, "How do we know we aren't brains in a vat?"  Time is money, so courts don't waste it on that.  What a relief!

The second great thing is that law must have explicit rules regarding all the little components of an evidence-based debate.

One of those components is the expert.  In law, an expert witness is basically anyone who has special knowledge which allows them to draw inferences that the average juror cannot.  Law sets the bar rather low for an expert; you only need to be above average in a particular area.

Expert witnesses are necessary because not everyone has the necessary information to make every decision in the practical world.  You might say, just gather the necessary information, but often there's so much necessary information that it would be too time-consuming (and thus money-consuming) to gather it all.  The worst part is that you may end up thinking you have sufficient information to draw the correct conclusion when you don't really.  The solution: experts.

I think a lot of skeptics don't like to rely to experts very much.  Why rely on someone else when you can be independent?  If you rely on someone else's knowledge, you have no way of knowing if they're wrong.  If you rely on someone else, you risk committing the argument from authority.

But perhaps we just need to get over it.  Experts are a necessary part of many debates.  They are on our side.  Skeptics are frequently framed as the group of people who dare to question authority, but I don't put to much stock in such narratives because we could just as easily be framed as the defenders of experts.

But if we always rely on experts, what's left for the lay skeptic to do, aside from knowing what the experts say?  In fact, there's still plenty of reasoning to be done.  There are lots of things that the pseudoscientific fringes say that are not directly covered by scientists. Usually, it's because it's too silly, and scientists don't want to waste time on it.  For example, when Creationists claim that evolution contradicts the second law of thermodynamics, that's just too stupid and doesn't deserve a penny of research funding.  When a ghosthunter claims to be able to find ghosts, obviously very few scientists are interested in testing that.

Other times, it's because the scientists are trained as scientists, not trained to counter the many rhetorical and political strategies of pseudoscience.  For instance, not every scientist knows the best way to respond to fear-mongering about cell phone radiation damaging our brains.  They just know how to test the idea, but apparently this isn't enough, since the negative results have not significantly dampened speculations.  This is the skeptic's area of expertise, because we are familiar with the behavior of pseudoscientific movements and organizations, and with the many ways that people are mislead.

(This post was partly influenced by Daniel Loxton's thoughts on skepticism's relation to science.)

10 comments:

Anonymous said...

Maybe you would be more cynical if you were on juries or just watched court sessions and saw some of the so-called "experts". There are many people who make their living as experts in trials. I think they slant their testimony to whichever side is paying them. And attorneys definitely go "expert shopping" to find experts that will take the side that they want. So experts on trials sometimes say utterly wrong things, even ridiculous things if no reasonable things can be said in their customer's favor. But if you are a potential juror and you appear too smart for one side's attorney, you will get dismissed and won't serve as a juror on that trial. And by the way, there are always experts saying diametrically opposing things on a trial. A knowledgeable person might know that 99% of real and honest experts would take one side, but it is not always clear to the jury that the split isn't 50-50.

Secret Squïrrel said...

@Anonymous: I was going to write about the same thing but you've said it so well! Instead, I'll give an example of a case that I followed but had no connection with, other than my slightly morbid interest.

A man was on trial for allegedly throwing his girlfriend to her death off a cliff. She was found some distance out from the foot of the cliff and had apparently landed head-first. A university physicist (an associate professor) was asked to investigate how she might have gotten there.

Long story-short, he determined that she was unlikely to have achieved the necessary launch speed by diving off. His experiments found that she could have achieved it by running and jumping off feet first, but since she landed head first and once she was in the air no forces would act upon her body to rotate it, she must have been thrown off by somebody (and like a spear - using two hands).

Now, I'm not a physicist, but I do know that objects can have preferential orientations when falling thru fluids, including air. Any skydiver will tell you that human feet have more drag than human heads. Her clothing may have also contributed to this.

Furthermore, if she did run up and then launch herself off at the edge, what's preventing her from imparting some angular momentum to herself such that she continues to rotate until she hits the rocks below?

The defence should have torn this witness to shreds.

The Barefoot Bum said...

There's nothing particularly philosophically interesting about our reliance on experts.

I don't have a lot of respect for philosophers or the profession of academic philosophy, but this lack of respect is based on the personal character of most philosophers, not on the discipline of philosophy itself. In the sense of thinking carefully about how we think about things, philosophy is a perfectly respectable endeavor.

As I noted earlier, "different" is an equivocal word. For example: in many senses you and I are different people. We are not only in different places, not only are we superficially physically different, we also have different attitudes, beliefs, characteristics, habits, desires, needs, and wants. And yet in another sense, we are the same: Not only do we share a lot of DNA in common, but there are a lot of things at varying levels of abstraction that we have in common.

I've noted elsewhere (and if you're interested in law and philosophy, I recommend Dagood's blog) that the "different ways of knowing" move is usually a bullshit equivocation fallacy, the equivalent of the argument that there are different kinds of automobiles, therefore my kitchen table is an automobile.

In law, the actual hard-core no-bullshit facts of the case consists of just the actual testimony, what the witnesses actually say while sitting on the witness stand. Everything else, especially the decision whether a witness is lying or mistaken, is a conclusion drawn from those facts.

In much the same sense, in physics the actual hard-core no-bullshit facts are what the scientist actually sees. When the voltmeter reads 9.2 v, the fact from which you're drawing conclusions is not that the voltage is 9.2 v, but that you see the volt-meter's reading of 9.2 v. You usually draw the conclusion that the simplest explanation for what you see when you look at the voltmeter is that the voltage really is 9.2 v.

You are basically using every instrument as an "expert" witness. It introduces testimony (its readings) for which the usual — but not invariable — conclusion is that reality corresponds to the testimony. If you are unsure of the reliability of an instrument, you introduce another instrument to see if the testimony is corroborated, and (because I know you're a good scientist) you introduce the new instrument that is most likely to not corroborate the original instrument. If you are uncertain about the voltmeter, you don't just introduce another voltmeter, you introduce an ammeter and an ohmmeter and make sure that A = V/Ohms.

Another essential similarity is that there is no algorithmic way to determine the truth, either in law or in science. There is no algorithm where you put in the evidence and out pops the truth in the same sense that there is an algorithm that will do long division. There's no way to take the skill and creativity out of the search for truth, and if computers can ever actually search for truth themselves, it will not be because we've found a truth-finding algorithm, but because we've found a way to make computers skillful and creative.

A scientist without skill or creativity is not going to find much scientific truth (except perhaps by pure dumb luck) regardless of what the truth actually is, and a lawyer without skill or creativity is not going to prevail at trial regardless of the underlying truth of the matter.

miller said...

Anonymous,

I agree that experts on trials can often mislead juries, for the multiple reasons you stated. However, I don't think we should conclude that experts are unnecessary (and I know you never said they were). I think it is a lesson in how to improve the quality of evidence we collect from experts.

In skepticism, we are perfectly allowed to set the bar much higher for experts. We're not limited to calling one expert at a time. We know better than to go expert shopping, and how to spot it when our adversaries go expert shopping. We have all these extra resources and tools at our disposal that are not possible in a court of law, so we should use them.

Barefoot Bum,

One of the underlying points of introducing law as another "way of knowing" is to explore the freedoms and constraints of a valid "way of knowing". You've explained some of the similarities between science and law; these suggest constraints. When I pointed out that law is much interested in philosophical questions, this suggests a freedom (though this freedom is only allowed because law does not purport to be an all-encompassing search for truth). Many of my readers have expressed cynicism about law; this suggests that we need still more constraints to have a more reliable way of knowing.

The hidden subtext is that when people suggest religion or spirituality as an alternate way of knowing, they have yet to show that it obeys sufficient constraints to even reach half the nobility of law.

DagoodS said...

miller,

I agree with the general principle being stated here; although as anonymous and secret squirrel both pointed out, it would fail if applied as performed in our courts today.

The rules of evidence are (ideally) designed to present the most reliable evidence, and weed out less credible or less dependable evidence. An item of concern is to limit eyewitnesses to only testify as to what they saw or heard. We do not want their opinion, speculation, or third-party information filtering into the pool of data. Therefore we limit a lay witness’ testifying to opinion (FRE 701).

For example, we only want the witness to say, “I saw a 50 meter skid mark on the road, ending at the crashed vehicle.” Explicitly limiting to what they actual saw. What we don’t want is “That car must have been going very fast, because it left a 50 meter skid mark.” That is opinion testimony the witness may not be qualified to make.

Skid marks vary depending on road curvature, surface condition, vehicle turning, tire condition, weather, etc. Since this is outside general knowledge, we would bring in an expert to inspect, test and experiment; rendering an opinion regarding the speed of the vehicle. (Again, as previously pointed out--due to range of precision--one side will hire an expert to testify to the lowest speed possible, whereas the other side will hire an expert to testify to the greatest speed possible. We call it “Battle of the Experts.”)

Another aspect helpful to your position is that experts (when used correctly) not only testify to their opinion, but help the trier-of-fact understand how they came to that conclusion, using all the facts both positive and negative. They do not merely testify, “The car was traveling at 100 – 120 kph at the beginning of the skid, and had reduced its speed to 25 – 30 kph at the end of the skid” and hop off the witness stand. We go through what testing they did, what they measured, what calculations they did, what they observed, etc. to come to their conclusion.

And because it is an adversarial system (and the other side has consulted with their own expert) the expert will be cross-examined regarding any incorrect data, incorrect measurements, etc. which would cause variation in his/her opinion. We hope this adversarial system will keep experts honest, albeit recognizing the imperfections in the system itself. (Why we have appellate courts.)

I can’t help but think of the Intelligent Design/Evolution debate…the “Battle of the Experts.” This is where (in my opinion) scientists have let down the ball. IDers understand they are talking to a “trier-of-fact” constituting the general public, and are explaining in helpful analogies, and simple terms how the measurements, data and experiments support their position. As a complete layperson, and generally unskilled in ways of science, I cannot tell you how many times I have read a scientist scoffing an IDer with what is (I am sure) a completely obvious explanation to the scientist but is far, far over my head. IDers are trying to convince a jury; scientists are trying to do science.

This is where your suggestion tends to break down. Experts are inhibited by the same limitations, biases and abilities as all humans. They are equally prone to be incorrect. Well-informed, better educated, yes. But still very human in their determinations.

Absent the adversarial ability to cross-examine, what is to prevent an expert from espousing any opinion? Especially when that generates substantial prestige and funds for the expert from the proponents? (*cough, cough* Dembski) Think about this, if you pay an expert $25,000 to testify—they have their money. All they need to do is keep you happy—not concern themselves with the truth. Without an adversarial system in place, we are reduced to examining the expert’s opinion as closely as another other claim.

The expert may be doing too much testifying to convince a jury, and not enough testifying about the actual results.

miller said...

Thank you for the explanation DagoodS.

My first reaction is to analogize skepticism to the practice of cross-examination and the adversarial system. Skeptics frequently "cross-examine" people like Dembski, Behe, etc. by scrutinizing their credentials and their reasoning. And since the real experts usually aren't motivated to talk to the public and participate in an adversary system, skeptics will seek out the experts and offer to popularize their opinions.

I'm wondering if this analogy makes any sense or if I'm just talking nonsense.

The Barefoot Bum said...

I think it would be helpful to be more specific about what we mean by "rely": there are varying degrees of "reliance".

One thing that I've really gained from studying philosophy for 10 years is to smell out equivocation, ambiguity and imprecision. Natural language is actually very poorly suited for very precise description, which is why scientists and engineers tend to rely on mathematics and jargon.

(Keep in mind that I have a Law & Order JD: I am not, as Dagood is, an actual expert in legal procedure or the rules of evidence.)

As Dagood notes, a court of law relies on expert opinion in a very different way it relies on lay testimony. In a philosophical sense, courts consider lay testimony more authoritative than expert opinion. If a lay person testifies, "I saw a 50 m skid mark," then the lawyer is entitled (under ordinary circumstances) to consider that there really was a 50 m skid mark to be a fact in evidence on the basis of no more than the witness's statement. Of course, a witness might be mistaken, deluded or lying. However, a lawyer who wishes to controvert a witness's testimony must introduce additional evidence

Whereas, as Dagood notes, an expert does not merely introduce an opinion directly into evidence; under ordinary circumstances the explanation and justification of that opinion needs to be introduced for the opinion to be taken as a fact in evidence.

The Barefoot Bum said...

This is where (in my opinion) scientists have let down the ball.

I think this something of an overstatement. As you note, scientists are primarily skilled in doing science, not as lawyers in persuading lay people (juries). To a large extent, most scientists have not "picked up the ball" in the first place or they have had it unwillingly thrust upon them.

Those scientists who have picked up the ball intentionally — notably Dawkins and the authors of Talk.Origins — usually do a very good job of explaining complicated scientific topics to a lay audience.

Scientists also suffer from a handicap that lawyers really don't have to deal with: in court, people cannot just flat-out lie through their teeth about easily verified facts, at least not without facing severe penalties, including prison. One of the problems with most science deniers and conspiracy theorists is that they can tell the most transparent and obvious lie without fear even of losing their credibility, much less of going to prison. This failure to enforce adherence to even the most basic level of factual truth is an enormous handicap.

DagoodS said...

Miller,

Of course you can analogize skeptical criticism to cross-examination…(one could analogize pumpkins to space rockets, I suppose) but I think it is closer to “nonsense” than “sense.” Sorry.

There are two reasons for this.

First, because the expert witness MUST answer. Both because the court will order it, and it is extremely detrimental to the expert’s credibility for him/her to avoid the question, especially when questioned by a skilled litigant. I could criticize a Dembski (since we are picking on IDers) with a blog-a-day, a forum, local letters, a facebook page, etc. And he could completely ignore me.

Second—more importantly—cross-examination takes place to point out issues to a neutral determinate. Here is where I, personally, believe the judicial adversary system is a good model for framing a debate. Too many arguments are made with the intention of convincing the already convinced. (I.e.—the entire field of Christian apologetics.)

Juries are neutral to the outcome of the trial. If the Plaintiff wins $1,000,000—the jurors neither pay the money, nor receive it. If the accused goes to jail or is set free—the jurors go home to the same dinner of pot roast and potatoes. Jurors, in my experience, are concerned about one thing: getting it right. They want the right person to be awarded the right amount of money. They want a guilty person to be found guilty, and fear an innocent person being imprisoned.

Therefore, when presenting a case, we are trying to convince a determinate that has no inherent bias either for/against our position, and will render a decision based on the facts, without benefit or detriment to themselves. This makes us carefully precise in our questions.

When we criticize Dembski—who listens? Basically three possible groups:

1) Non-neutral/pro-Dembski.
2) Non-neutral/anti-Dembski
3) Neutral.

The first two groups are essentially meaningless; the first will dismiss us as heretics, the second applaud our efforts regardless. The only people possibly effected is that third group, and they are so ill-defined, we may never know what convinces them or not.

Please understand, I think we DO need to continue to criticize IDers, vaccine-deniers, Christians, Muslims, homeopaths, chiropractors, etc. I’m just not sure “cross-examination” is a suitable analogy.

miller said...

Okay, I see.

Most of my knowledge of law comes exclusively from this class which sought to compare legal rules of evidence to those of other disciplines. The purpose, I think, was to understand the structure of an evidence-based debate. But I definitely see that there are some major differences between a court of law and a public debate.