The 95 Percentile Fraud

Home Page

My Latest Articles

Buying the Book

Reviews by readers

Reviews by critics

More about the book

About me

Email me

Statistics is a wonderful tool. It's been described as "a precise way to tell half-truths inaccurately". The bell-shaped curve, the so-called "normal curve" has features that really appeal to diligent groups devoted to "protecting" us from all possible hazards, both real and imaginary.

Assume that I'm about to market my new weight-loss elixir. "Use It and Lose It" (© Jay D Mann 2006) is a homeopathic extract of ancient minerals formed long before human contamination occurred. In other words, it's our local artesian-derived unchlorinated tap water. I'm not going to sell this product without a thorough test. So I've run a preliminary evaluation inside my computer using Resampling Statistics. Thirty subjects took the treatment for a binary month. As one might expect (and as I programmed into my computer), the average weight loss was zero, but there were deviations above and below that.

Therefore, I'm going to advertise "Lose up to two kilograms". That's a fair statement, so long as you disregard the equally likely outcome of gaining two kilograms. The most likely result is, of course, no weight change at all. (But think of how happy your kidneys and my bank balance will be from all the extra water you'll be drinking.)

Most people, and I suspect most consumer-protection agencies, would regard my misuse of such results as unethical and dishonest. My only defense would be that I'm merely copying the routine actions of the U.S. Environmental Protection Agency (EPA). As a New Zealand citizen, I would also cite the publications of our own NZ Ministry of the Environment.

You don't believe a reputable government agency, one with an $8 billion annual budget, would lie? There used to be a revealing webpage on the EPA website (www.epa.gov/safewater/mdbp/stmay99.htm) but that has disappeared. I offer instead my own copy of that webpage: EPA copy

My attention was first caught by their promise to save $45 million of medical treatment expenses at a cost of $701 million for water purification. Any agency that has the chutzpah to seriously talk about spending seven hundred million dollars in order to save forty-five million deserves serious respect and attention.

Some background: Chlorine is used to stop dangerous bugs developing in swimming pools. Some of the pool wastewater might get into drinking water. EPA high-dose studies with rodents fed chlorinated chemicals show a risk of bladder cancer. Using a controversial linear model, extrapolating cancer risk into the murky depths of super-low-concentration, EPA predicts an "upper bound" figure of 24 cancer cases per year from second-hand swimming pool water. That's 24 cases of cancer per year for the entire United States population!

But elsewhere in the EPA document it's revealed that the most likely value is not "24" but "0.2". That's right, at a cost of two-thirds of a billion dollars, one bladder cancer would (in theory) be prevented every five years. Since these numbers are not that precise, "0.2" is just an erudite way of saying "zero".

How did EPA turn zero danger into 24 Cancer Ward patients?

When I mention anything statistical to most people, their eyeballs roll up, their breathing becomes shallow, and they don't recover consciousness until I've finished my futile explanation. You're not that way, are you? So here is a non-mathematical explanation of what's going on.

Almost everything that we can measure winds up giving us a bell- shaped curve. The better the data, the less scatter, the sharper the curve. (Electrical people call this "high-Q".) Obviously there is a peak, a maximum, while the sides taper off above and below. The tighter the data, the narrower the peak. The width of the curve is described by the peak together with two magical points called "boundaries", "95 percent confidence limits", "95th percentile". The meaning is simple: if I repeated the measurements again and again, most of the time my new results would be close to the peak value. However, one time out of 40 the results would be above the "upper boundary", but not by very much.

There are various practical outcomes to this. If we're making aircraft passenger seats, we have to think about the occasional passenger who will be too wide across the bottom to fit into an average-sized seat. But if we're calculating how many passengers we can squeeze into our airplane, we will use the mean (average) value for butt size, and make a small correction for extra-wide people who'll take up two seats. No aircraft designer would make every seat extra-wide just to accommodate a small percentage of overweight passengers.

The EPA document clearly shows how they've misused the 95 percentile figure. Their own computers forecast a zero risk from swimming pool water, but because their forecasts have a large decree of uncertainty, there was an upper bound limit of 24. That means one chance of 40 that the true value might be more than 24. It also means an identical chance that traces of swimming pool water might avert 24 cases of bladder cancer. (The word for this is "hormesis", and it's gaining respectability.)

There's a slight-of-hand trick involved here. The EPA treats the least likely figure, the upper 95% value, as though it is the most likely value. That's how they calculate how many cancers are might be involved. But this is no more ethical than my advertising "lose up to two kilograms". If I have to admit to my customers that their most likely weight loss is zero, then the EPA admit that the most likely risk from usesd swimming pool water is zero. Of course, it's hard to justify a multi-billion dollar budget to prevent non-risks!

EPA has used the upper-bound approach on all sorts of purported environmental issues. The courts seem to have allowed this as being a matter of administrative detail. But courts are not famous for being numerically apt. The EPA defends its use as "a very conservative application and … quite protective". It points out "the upper bound estimate is usually within two orders of magnitude of the maximum dose estimate". That's another attempt, I suspect, to pull the wool over our eyes. "Two orders of magnitude" means a factor of one hundred! That's what happens in this swimming pool caper: the most likely number is 0.2 and the upper bound risk is 24, slightly more than one hundred times greater. Frankly if I had a forecast with a hundred-fold error range in both directions, I'd slink off in acute embarrassment. (Note added 1 November 2006: EPA has, in fact, moved this report out of their web page! I'll try to track it down.)

The fallacy of faking a risk by using the upper bound figure as though it's real is clearly shown in the EPA's recommendations for water treatment, a wonderful demonstration that if you start with incorrect assumptions, you wind up with bizarre conclusions. Their proposal would avert, at most, 24 cases of bladder cancer per year. That compares with 63 thousand new bladder cancer cases diagnosed in the U.S. every year. It happens that half these cases are linked with tobacco smoking. I suspect that $701 million spent in anti-smoking efforts would save far more than 24 lives.

About two-thirds of the bladder cancer patients can be cured if detected early. How many early-detection tests would $701 million pay for? The EPA mathematics are, it turns out, not so much "protective" and "conservative" as a total waste of money that could otherwise have gone into genuinely useful strategies to reduce the death toll from bladder cancer.

So far as I know, the ridiculous EPA proposals on drinking water haven't been implemented. How many other actions, also based on the fallacious upper-bound methods, have slipped through because no one has picked up this statistical trickery. I regard it as disgusting, dishonest and disreputable, but that's only my opinion. I'd like to see EPA (and other government agencies) be required to calculate cost-benefit relationships, especially when alternative ways to ameliorate a problem are conceivable.

© Jay D Mann, November 2006