Lord's paradox

Short description: Statistical paradox

In statistics, Lord's paradox raises the issue of when it is appropriate to control for baseline status. In three papers, Frederic M. Lord gave examples when statisticians could reach different conclusions depending on whether they adjust for pre-existing differences. Holland & Rubin (1983) use these examples to illustrate how there may be multiple valid descriptive comparisons in the data, but causal conclusions require an underlying (untestable) causal model. Pearl used these examples to illustrate how graphical causal models resolve the issue of when control for baseline status is appropriate.

Lord's formulation

Sketch of Lord's paradox.

The most famous formulation of Lord's paradox comes from his 1967 paper:

“A large university is interested in investigating the effects on the students of the diet provided in the university dining halls and any sex differences in these effects. Various types of data are gathered. In particular, the weight of each student at the time of his arrival in September and his weight the following June are recorded.” (Lord 1967, p. 304)

In both September and June, the overall distribution of male weights is the same, although individuals' weights have changed, and likewise for the distribution of female weights.

Lord imagines two statisticians who use different common statistical methods but reach opposite conclusions.

One statistician does not adjust for initial weight, instead using t-test and comparing gain scores (individuals' average final weight − average initial weight) as the outcome. The first statistician claims no significant difference between genders: "[A]s far as these data are concerned, there is no evidence of any interesting effect of diet (or of anything else) on student weights. In particular, there is no evidence of any differential effect on the two sexes, since neither group shows any systematic change." (pg. 305) Visually, the first statistician sees that neither group mean ('A' and 'B') has changed, and concludes that the new diet had no causal impact.

The second statistician adjusts for initial weight, using analysis of covariance (ANCOVA), and compares (adjusted) final weights as the outcome. He finds a significant difference between the two dining halls. Visually, the second statistician fits a regression model (green dotted lines), finds that the intercept differs for boys vs girls, and concludes that the new diet had a larger impact for males.

Lord concluded: "there simply is no logical or statistical procedure that can be counted on to make proper allowance for uncontrolled preexisting differences between groups."

Responses

There have been many attempts and interpretations of the paradox, along with its relationship to other statistical paradoxes. While initially framed as a paradox, later authors have used the example to clarify the importance of untestable assumptions in causal inference.

Importance of modeling assumptions

Bock (1975)

Bock responded to the paradox by positing that both statisticians in the scenario are correct once the question being asked is clarified. The first statistician (who compares group means and distributions) is asking "are there differences in average weight gain?", whereas the second is asking "what are the differences in individual weight gain?"

Cox and McCullagh (1982)

Cox and McCullagh interpret the problem by constructing a model of what could have happened had the students not dined in the dining hall, where they assume that a student's weight would have stayed constant. They conclude that in fact the first statistician was right when asking about group differences, while the second was right when asking about the effect on an individual.

Holland and Rubin (1983)

Holland & Rubin (1983) argue that both statisticians have captured accurate descriptive features of the data: Statistician 1 accurately finds no difference in relative weight changes across the two genders, while Statistician 2 accurately finds a larger average weight gain for boys conditional on a boy and girl have the same starting weight. However, when turning these descriptions into causal statements, they implicitly assert that weight would have otherwise stayed constant (Statistician 1) or that it would have followed the posited linear model (Statistician 2).

“In summary, we believe that the following views resolve Lord's Paradox. If both statisticians made only descriptive statements, they would both be correct. Statistician 1 makes their unconditional descriptive statements that the average weight gains for males and females are equal; Statistician 2 makes the conditional (on X) statement that for males and females of equal September weight, the males gain more than the females. In contrast, if the statisticians turned these descriptive statements into causal statements, neither would be correct or incorrect because untestable assumptions determine the correctness of causal statements... Statistician 1 is wrong because he makes a causal statement without specifying the assumption needed to make it true. Statistician 2 is more cautious, since he makes only a descriptive statement. However, unless he too makes further assumptions, his descriptive statement is completely irrelevant to the campus dietician's interest in the effect of the dining hall diet." (pg. 19)

Moreover, the underlying assumptions necessary to turn descriptive statements into causal statements are untestable. Unlike descriptive statements (e.g. "the average height in the US is X"), causal statements involve a comparison between what happened and what would have happened absent an intervention. The latter is unobservable in the real world, a fact that Holland & Rubin term "the fundamental problem of causal inference" (pg. 10). This is explains why researchers often turn to experiments: while we still never observe both counterfactuals for a single subject, experiments let us make statistical claims about these differences in the population under minimal assumptions. Absent an experiment, modelers should carefully describe the model they use to make causal statements and justify those models as strongly as possible.

Pearl (2016)

Pearl (2016) agrees with Lord’s conclusion that the answer cannot be found in the data, but he finds Holland and Rubin’s account to be incomplete. In his views, a complete resolution of the Paradox should provide an answer to Lord’s essential question: "How to allow for preexisting differences between groups?" Moreover, since the answer depends on the causal model assumed, we should explain: (1) Why people find Lord’s story to be "Paradoxical" rather than "In need of more information" and, (2) How to properly utilize causal models to answer Lord’s question, regardless of whether they are testable or not.

To this end, Pearl used a simplified version of Lord’s Paradox, proposed by Wainer and Brown, in which gender differences are not considered. Instead, the quantity of interest is the effect of diet on weight gain, as shown in Figure 2(a).

Revised version of Lord’s paradox and its causal diagram (from )

The two ellipses represent two dining halls, each serving a different diet, and each point represents a student's initial and final weights. Note that students who weigh more in the beginning tend to eat in dining hall B, while the ones who weigh less eat in dining hall A. The first statistician claims that switching from Diet A to B would have no effect on weight gain, since the gain W_F – W_I has the same distribution in both ellipses. The second statistician compares the final weights under Diet A to those of Diet B for a group of students with same initial weight W₀ and finds that latter is larger than the former in every level of W₀. He concludes therefore that the students on Diet B gain more than those on Diet A. As before, the data can’t tell us whom to believe, and a causal model must be assumed to settle the issue. One plausible model is shown in Figure 2(b). In this model, W_I is the only confounder of D and W_F, so controlling for W_I is essential for deconfounding the causal effect needed. Assuming this model, the second statistician would be correct and the first statistician would be wrong.

This analysis also unveils why Lord’s story appears paradoxical, and why generations of statisticians have found it perplexing.

According to Pearl, the data triggers a clash between two strong intuitions, both are valid in causal thinking, but not in the non-causal thinking invoked by the first statistician.

One intuition claims that, to get the needed effect, we must make “proper allowances” for uncontrolled preexisting differences between groups” (i.e. initial weights). The second claims that the overall effect (of Diet on Gain) is just the average of the stratum-specific effects. The two intuitions are valid, but seem to clash when we interpret the first statistician’s finding as zero effect when, in fact, his finding merely entails equality of distributions, and says nothing about ``effects. This can also be seen from Figure 2(b), which allows D to causally affect Y while, simultaneously, be statistically independent of it (due to path cancelations).

This resolution of Lord’s Paradox answers both questions: (1) How to allow for preexisting differences between groups and (2) Why the data appear paradoxical. Pearl's do-calculus further answers question (1) for any causal model assumed, including models with multiple unobserved confounders.

Initial weight as a mediator

Going back to Lord's original problem of comparing boys and girls, Pearl (2016) posits another causal model where sex and initial weight both influence the final weight. Moreover, since sex also influences the initial weight, Initial Weight becomes a mediating variable: sex influences final weight both through a direct effect and an indirect effect (by influencing initial weight, which then influences final weight). Note that none of these variables are confounders, so controls are not strictly necessary in this model. However, the choice of whether to control for initial weight dictates which effect the researcher is measuring: the first statistician does not control and measures a total effect, while the second does control and measures a direct effect.

"Cases where total and direct effects differ in sign are commonplace. For example, we are not at all surprised when smallpox inoculation carries risks of fatal reaction, yet reduces overall mortality by eradicating smallpox. The direct effect (fatal reaction) in this case is negative for every stratum of the population, yet the total effect (on mortality) is positive for the population as a whole." (pg 4)

Tu, Gunnell, and Gilthorpe (2008) use a similar causal framework, but counter that the conceptualization of direct and total effect is not the best framework in many cases because there are many different variables that could be controlled for, without an experimental basis that these are separate causal paths.

Relation to other paradoxes

According to Tu, Gunnell, and Gilthorpe, Lord's paradox is the continuous version of Simpson's paradox. Those authors state that Lord's Paradox, Simpson's Paradox, and the suppression of covariates by uncorrelated predictor variables are all the same thing, namely a reversal paradox.

Importance

Broadly, the "fundamental problem of causal inference" and related aggregation concepts Simpson's paradox play major roles in applied statistics. Lord's Paradox and associated analyses provide a powerful teaching tool to understand these fundamental statistical concepts.

More directly, Lord's Paradox may have implications for both education and health policies that attempt to reward educators or hospitals for the improvements that their children/patients made under their care, which is the basis for No Child Left Behind. It has also been suspected to be at work in studies linking IQ to eye defects.