Vuong's closeness test

In statistics, the Vuong closeness test is a likelihood-ratio-based test for model selection using the Kullback–Leibler information criterion. This statistic makes probabilistic statements about two models. They can be nested, strictly non-nested or partially non-nested (also called overlapping). The statistic tests the null hypothesis that the two models are equally close to the true data generating process, against the alternative that one model is closer. It cannot make any decision whether the "closer" model is the true model.

Technical description

With strictly non-nested models and iid exogenous variables, model 1 (2) is preferred with significance level α, if the z statistic

[math]\displaystyle{ Z=\frac{ \operatorname{LR}_N(\beta_{ML,1},\beta_{ML,2})} {\sqrt{N}\omega_N} }[/math]

with

[math]\displaystyle{ \operatorname{LR}_N(\beta_{ML,1},\beta_{ML,2}) = L^1_N-L^2_N-\frac{K_1-K_2} {2} \log N }[/math]

exceeds the positive (falls below the negative) (1 − α)-quantile of the standard normal distribution. Here K₁ and K₂ are the numbers of parameters in models 1 and 2 respectively.

The numerator is the difference between the maximum likelihoods of the two models, corrected for the number of coefficients analogous to the BIC, the term in the denominator of the expression for Z, [math]\displaystyle{ \omega_N \, }[/math], is defined by setting [math]\displaystyle{ \omega_N^2 }[/math] equal to either the mean of the squares of the pointwise log-likelihood ratios [math]\displaystyle{ \ell_i\, }[/math], or to the sample variance of these values, where

[math]\displaystyle{ \ell_i = \log\frac{f_1(y_i\mid x_i,\beta_{ML,1})}{f_2(y_i\mid x_i,\beta_{ML,2})}. }[/math]

For nested or partially non-nested (overlapping) models the statistic

[math]\displaystyle{ 2\operatorname{LR}_N(\beta_{ML,1},\beta_{ML,2})\, }[/math]

has to be compared to critical values from a weighted sum of chi squared distributions. This can be approximated by a gamma distribution (in shape-rate form):

[math]\displaystyle{ M_m(\cdot,\mathbf\lambda)\sim \Gamma(b,p)\, }[/math]

with

[math]\displaystyle{ \mathbf\lambda=(\lambda_1, \lambda_2, \dots, \lambda_m),\, }[/math]

[math]\displaystyle{ m=K_1+K_2,\ b=\frac 1 2 \frac {\left(\sum\lambda_i\right)^2} {\sum\lambda_i^2} }[/math]

and

[math]\displaystyle{ p=\frac 1 2 \frac {\sum\lambda_i} {\sum\lambda_i^2}. }[/math]

[math]\displaystyle{ \mathbf\lambda }[/math] is a vector of eigenvalues of a matrix of conditional expectations. The computation is quite difficult, so that in the overlapping and nested case many authors only derive statements from a subjective evaluation of the Z statistic (is it subjectively "big enough" to accept my hypothesis?).

Improper use for zero-inflated models

Vuong's test for non-nested models has been used in model selection to compare a zero-inflated count model to its non-zero-inflated counterpart (e.g., zero-inflated Poisson model versus ordinary Poisson model). Wilson (2015) argues that such use of Vuong's test is invalid as a non-zero-inflated model is neither strictly non-nested nor partially non-nested in its zero-inflated counterpart. The core of the misunderstanding appears to be the terminology, which offers itself to being incorrectly understood to imply that all pairs of non-nested models are either strictly non-nested or partially non-nested (aka overlapping). Crucially, the definitions of strictly non-nested and partially non-nested in Vuong (1989) do not unite to mean "all pairs of models that are not nested". In other words, there are non-nested models that are neither strictly non-nested nor partially non-nested. The zero-inflated Poisson model and its non-zero-inflated counterpart are an example of such a pair of non-nested models. Consequently, Vuong's test is not a valid test for discriminating between them.

Example of strictly and partially non-nested models

Vuong (1989) gives two examples of strictly non-nested models:

• A pair of standard linear regression models with different distributional assumptions on the distribution of error terms (e.g., normally distributed and logistically distributed).
• A pair of standard linear regression models with the same distributional assumptions on the distribution of error terms but different functional forms such as [math]\displaystyle{ Y_t=\theta_1+\theta_2'Z_t+\epsilon_t^A }[/math] and [math]\displaystyle{ Y_t=\exp(\gamma_1+\gamma_2'Z_t)+\epsilon_t^B }[/math], where [math]\displaystyle{ \theta_2\neq0, \gamma_2\neq0 }[/math] and [math]\displaystyle{ Z_t }[/math] is a non-degenerate real random vector.

Vuong (1989) also gives an intuitive example of partially non-nested (aka overlapping) models:

• A pair of standard linear regression models with some common explanatory variables and neither model nested in the other.

References

• Vuong, Quang H. (1989). "Likelihood Ratio Tests for Model Selection and non-nested Hypotheses". Econometrica 57 (2): 307–333. doi:10.2307/1912557. https://authors.library.caltech.edu/81424/1/sswp605.pdf. 

• Genius, Margarita; Strazzera, Elisabetta (2002). "A note about model selection and tests for non-nested contingent valuation models". Economics Letters 74 (3): 363–370. doi:10.1016/S0165-1765(01)00566-3. 

• Wilson, Paul (2015). "The Misuse of The Vuong Test For Non-Nested Models to Test for Zero-Inflation". Economics Letters 127 (2): 151–153. doi:10.1016/j.econlet.2014.12.029.