Policy Making
20. Size Matters: Showing, Quantitatively, How Much

The Tinbergen model hides another important input to the production of expert policy advice: using statistics to make scientific inferences. Page through the annals of any agency or think tank related to fiscal or monetary policy, such as The Economic Report of the President or the latest report issued by the Urban Institute or the American Enterprise Institute, and you will find table after table of statistical data and a host of economic claims backed up by statistical models and reasoning. The more academic and technical the intended audience, the more explicit will be the statistical methodology revealed in the published report.

One statistical technique in particular has come to dominate empirical economic research, namely, "statistical significance" in the context of "regression analysis." Soon you will be introduced to tests of statistical significance and regression analysis in an elementary statistics course or, if you haven't already. Or, should you should choose to go on in your pursuit of economic knowledge, you'll become intimate with tests of "statistical significance" in a first course on "econometrics."

Statistical significance is the output of a hypothesis test. It tells the economist how likely the hypothesized zero-effect of variable X on variable Y is to be wrong over a large imaginary repetition of random trials. 5 percent error or less is typically said to be "significant": that is, good enough to say the observed effect between X and Y is "permanent" or "real." Statistical significance does not tell anything about economic significance.

Regression analysis uses quantitative data to estimate relationships between economic variables, such as disposable income on consumption expenditures, or welfare subsidies on the labor supply of the poor. Keynesians hypothesize a large, non-zero effect between income and consumption and a negligible or zero-effect of welfare subsidies on labor supply.

Econometrics is the estimation and testing of economic relationships aided by various methods of statistical inference, especially regression analysis and tests of statistical significance.

Counting is crucial to a policy science, including economic science. But often enough you can't afford to count everything. If you don't have your hands on every single data point in your population, then sampling theory or some other statistical methodology is often the best available way to estimate them. As we've seen over and over again throughout this book-for example, in the early micro chapter on income statements and balance sheets and in the macro chapter on the multiplier/accelerator model-counting is a crucial input for answering the tough scientific question, "size matters/how much?" In recent decades economists have misused their tests of statistical significance and regressions. And not only economists: scientists in all the life and social sciences have misused the test. (Ziliak and McCloskey, Size Matters: How Some Sciences Lost Interest in Magnitude, and What to Do About It [University of Michigan Press, 2007].)

They have stopped asking the size matters/how much question, as if they didn't care how much liquid money was necessary to stimulate aggregate demand, or how much of an increase in taxes would be sufficient to bring the economic down to its knees.

Decades ago many (but not all) economists began to ask a wrong question of their statistical tool, "statistical significance." "Does an effect exist?" they asked. "Is it statistically significant?"-meaning, is it "precise" in the one narrow sense that such a test accounts for precision. "Yes or no" they'd reply. "Is there any effect present?" is the qualitative question they began to substitute for the properly quantitative question, "What is the size of the effect?" or, what amounts to the same, "So what?" What matters to the autonomous spending multiplier or the negative income tax or the minimum wage law is the size of the effect and the social meaning of that size in the wider political economy. Finding a low probability of being wrong about an unimportant effect is not useful to policy makers or society. Economic advice that is informed by test after test of statistical significance and not by the truly important test-the test of economic significance-is going to be correct only by chance.

Sir Alec Cairncross (1911-1998) was an economic historian and professor of economics of legendary common sense and learning, an economic consultant greatly admired in British and international circles. His dissertation was a pioneering piece of "cliometrics"-that is, of quantitative economic history. When Cairncross was alive, The Cairncross Criterion of a statistical model was: "Would Sir Alec be willing to take this model to Riyadh?" That is, would Sir Alec use it for advising on real economic development in real places such as Saudi Arabia, Nigeria, or the United States? Economic policy advisors should revive the Cairncross Criterion. Significance testing is not what Sir Alec had in mind. He reported of his experience at HM Treasury after the War that his fellow economists, fresh from the first courses in econometrics, would give one significant estimate of the elasticity of demand for British exports in the morning, and quite another significant one by evening. Cairncross didn't believe either.

Caption: "The Cairncross Criterion" required an answer to the Size matters/ How much? Question. Alec Cairncross was a highly respected economic advisor of the 20th century. He was not impressed by statistical significance (or the lack of it). He sought-and deliberated over-magnitudes of economic significance.

Interestingly, the same Jan Tinbergen, despite his astonishing mathematical ingenuity, contributed to the misuse of statistical significance. But Tinbergen had the unusual common sense to examine a wide variety of empirical evidence before claiming he had "discovered something" about the economy; to Tinbergen, significance testing was just one of thirteen different ways of looking at an economic event. Economic policy is hard. But it would improve if it downplayed significance testing and instead followed the model of Sir Alec, focusing on matters of economic significance. Oomph-effect size-is what policy makers and citizens want from our statistical models of the economy. Precision is nice-sometimes we really do need to know the odds of being wrong-but oomph is the bomb. Knowing "how much" is crucial.