Criticisms of econometrics

There have been many criticisms of econometrics' usefulness as a discipline and perceived widespread methodological shortcomings in econometric modelling practices.

Difficulties in model specification

Like other forms of statistical analysis, badly specified econometric models may show a spurious correlation where two variables are correlated but causally unrelated. Economist Ronald Coase is widely reported to have said "if you torture the data long enough it will confess". McCloskey argues that in published econometric work, economists often fail to use economic reasoning for including or excluding variables, equivocate statistical significance with substantial significance, and fail to report the power of their findings.

Economic variables are not readily isolated for experimental testing, but Edward Leamer argues that there is no essential difference between econometric analysis and randomized trials or controlled trials provided the use of statistical techniques reduces the specification bias, the effects of collinearity between the variables, to the same order as the uncertainty due to the sample size.

Economists are often faced with a high number of often highly collinear potential explanatory variables, leaving researcher bias to play an important role in their selection. Leamer argues that economists can mitigate this by running statistical tests with different specified models and discarding any inferences which prove to be "fragile", concluding that "professionals ... properly withhold belief until an inference can be shown to be adequately insensitive to the choice of assumptions". However, as Sala-I-Martin showed, it is often the case that you can specify two models suggesting contrary relation between two variables. The phenomenon was labeled 'emerging recalcitrant result' phenomenon by Robert Goldfarb.

Lucas critique

Robert Lucas criticised the use of overly simplistic econometric models of the macroeconomy to predict the implications of economic policy, arguing that the structural relationships observed in historical models break down if decision makers adjust their preferences to reflect policy changes. Lucas argued that policy conclusions drawn from contemporary large-scale macroeconometric models were invalid as economic actors would change their expectations of the future and adjust their behaviour accordingly.

Lucas argued a good macroeconometric model should incorporate microfoundations to model the effects of policy change, with equations representing economic representative agents responding to economic changes based on rational expectations of the future; implying their pattern of behaviour might be quite different if economic policy changed.

Modern complex econometric models tend to be designed with the Lucas critique and rational expectations in mind, but Robert Solow argued that some of these modern dynamic stochastic general equilibrium models were no better as the assumptions they made about economic behaviour at the micro level were "generally phony".

Other mainstream critiques

Looking primarily at macroeconomics, Lawrence Summers has criticized econometric formalism, arguing that "the empirical facts of which we are most confident and which provide the most secure basis for theory are those that require the least sophisticated statistical analysis to perceive." He looks at two highly praised macroeconometric studies (Hansen & Singleton (1982, 1983), and Bernanke (1986)), and argues that while both make brilliant use of econometric methods, both papers do not really prove anything that future theory can build on. Noting that in the natural sciences, "investigators rush to check out the validity of claims made by rival laboratories and then build on them," Summers points out that this rarely happen in economics, which to him is a result of the fact that "the results [of econometric studies] are rarely an important input to theory creation or the evolution of professional opinion more generally." To Summers:

Successful empirical research has been characterized by attempts to gauge the strength of associations rather than to estimate structural parameters, verbal characterizations of how causal relations might operate rather than explicit mathematical models, and the skillful use of carefully chosen natural experiments rather than sophisticated statistical technique to achieve identification.

Austrian School critique

The current-day Austrian School of Economics typically rejects much of econometric modeling. The historical data used to make econometric models, they claim, represents behavior under circumstances idiosyncratic to the past; thus econometric models show correlational, not causal, relationships. Econometricians have addressed this criticism by adopting quasi-experimental methodologies. Austrian school economists remain skeptical of these corrected models, continuing in their belief that statistical methods are unsuited for the social sciences.

The Austrian School holds that the counterfactual must be known for a causal relationship to be established. The changes due to the counterfactual could then be extracted from the observed changes, leaving only the changes caused by the variable. Meeting this critique is very challenging since "there is no dependable method for ascertaining the uniquely correct counterfactual" for historical data. For non-historical data, the Austrian critique is met with randomized controlled trials. In randomized controlled trials, the control group acts as the counterfactual since they experience, on average, what the treatment group would have experienced had they not been treated. It is on this sound basis that parametric statistics (in the Gaussian sense) is based. Randomized controlled trials must be purposefully prepared, which historical data is not. The use of randomized controlled trials is becoming more common in social science research. In the United States, for example, the Education Sciences Reform Act of 2002 made funding for education research contingent on scientific validity defined in part as "experimental designs using random assignment, when feasible." In answering questions of causation, parametric statistics only addresses the Austrian critique in randomized controlled trials.

If the data is not from a randomized controlled trial, econometricians meet the Austrian critique with quasi-experimental methodologies. These methodologies attempt to extract the counterfactual post-hoc so that the use of the tools of parametric statistics is justified. Since parametric statistics depends on any observation following a Gaussian distribution, which is only guaranteed by the central limit theorem in a randomization methodology, the use of tools such as the confidence interval will be outside of their specification: the amount of selection bias will always be unknown. A better approximation to a randomized controlled trial provided by a quasi-experimental method will reduce this selection bias, but these methods are not rigorous, and one cannot deduce precisely how incorrect the familiar parametric measures such as power and statistical significance will be if they are calculated on these additional assumptions. When parametric statistics are used beyond their specifications, Econometricians argue that the insight will exceed the inaccuracy while Austrians argue that the inaccuracy will exceed the insight. A historical example of this debate is the Friesh–Leontief "Pitfalls" debate, with Friesh holding the Austrian position and Leontief holding the econometric position. Structural causal modeling, which attempts to formalize the limitations of quasi-experimental methods from a causality perspective, allowing experimenters to precisely quantify the risks of quasi-experimental research, is an emerging discipline originating with the work of Judea Pearl.

See also

• Econometrics
• Methodology of econometrics
• Macroeconomic model
• History of macroeconomic thought