Two-step M-estimators

Two-step M-estimators deals with M-estimation problems that require preliminary estimation to obtain the parameter of interest. Two-step M-estimation is different from usual M-estimation problem because asymptotic distribution of the second-step estimator generally depends on the first-step estimator. Accounting for this change in asymptotic distribution is important for valid inference.

Description

The class of two-step M-estimators includes Heckman's sample selection estimator, weighted non-linear least squares, and ordinary least squares with generated regressors.

To fix idea, let [math]\displaystyle{ \{W_{i}\}^n_{i=1} }[/math] [math]\displaystyle{ \subseteq R^d }[/math] be an i.i.d. sample. [math]\displaystyle{ \Theta }[/math] and [math]\displaystyle{ \Gamma }[/math] are subsets of Euclidean spaces [math]\displaystyle{ R^p }[/math] and [math]\displaystyle{ R^q }[/math], respectively. Given a function [math]\displaystyle{ m(;;;): R^d \times \Theta \times \Gamma\rightarrow R }[/math] , two-step M-estimator [math]\displaystyle{ \hat\Theta }[/math] is defined as:

[math]\displaystyle{ \hat \Theta:=\arg\max_{\Theta\epsilon\Theta}\frac{1}{n}\sum_{i}m\bigl(W_{i},\theta,\hat\gamma\bigr) }[/math]

where [math]\displaystyle{ \hat\gamma }[/math] is a parameter that needs to be estimated in the first step.

Consistency of two-step M-estimators can be verified by checking consistency conditions for usual M-estimators, although some modification might be necessary. In practice, the important condition to check is identification condition. If [math]\displaystyle{ \hat\gamma\rightarrow\gamma^*, }[/math] where [math]\displaystyle{ y^* }[/math]where [math]\displaystyle{ \gamma^* }[/math] is a non-random vector, then the identification condition is that [math]\displaystyle{ E[m(W_{1},\theta,\gamma^*)] }[/math] has a unique maximizer over [math]\displaystyle{ \Theta }[/math].

Under regularity conditions, two-step M-estimators have asymptotic normality. An important point to note is that asymptotic variance of a two-step M-estimator is generally not the same as that of the usual M-estimator in which the first step estimation is not necessary. This fact is intuitive because [math]\displaystyle{ \gamma^* }[/math] is a random object and its variability should influence the estimation of [math]\displaystyle{ \Theta }[/math]. However, there exists a special case in which the asymptotic variance of two-step M-estimator takes the form as if there were no first-step estimation procedure. Such special case occurs if:

[math]\displaystyle{ E \frac{\partial}{\partial\theta\partial\gamma}m(W_{1},\theta_{0},\gamma^*)=0 }[/math]

where [math]\displaystyle{ \theta_{0} }[/math] is the true value of [math]\displaystyle{ \theta }[/math] and [math]\displaystyle{ \gamma^* }[/math] is the probability limit of [math]\displaystyle{ \hat\gamma }[/math]. To interpret this condition, first note that under regularity conditions, [math]\displaystyle{ E \frac{\partial}{\partial\theta\partial\gamma}m(W_{1},\theta_{0},\gamma^*)=0 }[/math] since [math]\displaystyle{ \theta_{0} }[/math] is the maximizer of [math]\displaystyle{ E m(W_{1},\theta \gamma^*) }[/math]. So the condition above implies that small perturbation in γ has no impact on the First-Order condition. Thus, in large sample, variability of [math]\displaystyle{ \hat\gamma }[/math] does not affect the argmax of the objective function, which explains invariant property of asymptotic variance. Of course, this result is valid only as the sample size tends to infinity, so the finite-sample property could be quite different.

Involving MLE

Two-step M-estimator involving Maximum Likelihood Estimator is a special case of general two-step M-estimator. Thus, consistency and asymptotic normality of the estimator follows from the general result on two-step M-estimators.

When the first step estimation is MLE, under some assumptions, two-step M-estimator is more efficient [i.e. has smaller asymptotic variance] than M-estimator with known first-step parameter.

Let {V_i,W_i,Z_i}ni=1 be a random sample and the second-step M-estimator [math]\displaystyle{ \widehat{\theta} }[/math] is the following:

[math]\displaystyle{ \widehat{\theta} }[/math] ≔ [math]\displaystyle{ \underset{\theta\in\Theta}{\operatorname{arg\max}}\sum_{i}m(\,v_i,w_i,z_i: \theta\,,\widehat{\gamma }) }[/math]

where [math]\displaystyle{ \widehat{\gamma } }[/math] is the parameter estimated by ML procedure in the first step. For the MLE,

[math]\displaystyle{ \widehat{\gamma } }[/math] ≔ [math]\displaystyle{ \underset{\gamma\in\Gamma}{\operatorname{arg\max}}\sum_{i}\log f(v_{it} : z_{i} , \gamma) }[/math]

where f is the conditional density of V given Z. Now, suppose that given Z, V is conditionally independent of W. This assumption is called conditional independence assumption or selection on observables. Intuitively, this condition means that Z is a good predictor of V so that once conditioned on Z, V has no systematic dependence on W. Under the conditional independence assumption, the asymptotic variance of the two-step estimator is:

E[∇_θ s(θ₀,γ₀)]⁻¹ E[g(θ₀,γ₀ )g(θ₀,γ₀ )']E[∇_θ s(θ₀,γ₀)]⁻¹

where g(θ,γ) ≔ s(θ,γ)-E[ s(θ , γ) ∇_γ d(γ)^' ]E[∇_γ d(γ) ∇_γ d(γ)^' ]⁻¹ d(γ),

s(θ,γ) ≔ ∇_θ m(V, W, Z: θ, γ) , d(γ) ≔ ∇_γ log f (V : Z, γ), and ∇ represents partial derivative with respect to a row vector. In the case where γ₀ is known, the asymptotic variance is E[∇_θ s(θ₀,γ₀)]⁻¹ E[s(θ₀,γ₀ )s(θ₀,γ₀ )^']E[∇_θ s(θ₀,γ₀)]⁻¹ and therefore, unless E[ s(θ, γ) ∇_γ d(γ)^' ]=0, the two-step M-estimator is more efficient than the usual M-estimator. This fact suggests that even when γ₀ is known a priori, there is efficiency gain by estimating γ by MLE. An application of this result can be found, for example, in treatment effect estimation.