Statistical Significance

If you promise not to spread it around, I’ll tell you a secret about one of the main techniques of modern economics, the “test of statistical significance.” It runs much of modern economics, finance, sociology, psychology, and medicine.

It’s hard to explain. When I was being examined in Harvard graduate school back in 1966, I was asked to explain it. I couldn’t. I had by then taken three courses in econometrics, and had mastered all sorts of impressive proofs. I missed the goal kick by three meters.

Let me try.

We can’t ask every Brazilian every month whether she’s employed or not. Too expensive. So we ask a random sample of Brazilians, maybe 1,000, and apply the result to the nation. To a non-quantitative person it already sounds crazy. But I’ve had three terms of econometrics, and anyway I love numbers.

Is the estimate correct? The economist at the Ministry of Labor replies, “Of course it is. The sample size is 1,000, and so there is a very, very low probability that the estimate is off by much on account of sampling error. The estimate,” he concludes with a self-satisfied smirk, “is statistically significant.”

Set aside certain advanced quantitative worries of whether the sample is actually random or whether the rounding error in the computer programs are producing bad results. The fundamental problem is that the economist thinks that such a calculation shows the unemployment rate is important, that it matters. Yet accuracy in thus narrow sense does not mean anything of the kind. For importance, for mattering, we have to answer a philosophical question of what unemployment means, and what size of it would cause mattering.

Yes, I know. Hard to understand. So hard that economics and statistical medicine don’t.

One more attempt, and then I’ll leave you to look into the “significance-test controversy” in economic and medical statistics. Google David Rothman or Stephen Ziliak, for example.

You ask me whether it’s a nice day in Chicago. I reply “Six.” You say, “What does that mean?” I reply angrily, “What do you mean, ‘mean.’ It’s six. Don’t you believe in numbers?!”

Numbers, no more than words, do not come with their own meaning. Economic and medical researchers think they do. It’s a scientific disaster.