Empirical likelihood based modal regression

Abstract

In this paper, we consider how to yield a robust empirical likelihood estimation for regression models. After introducing modal regression, we propose a novel empirical likelihood method based on modal regression estimation equations, which has the merits of both robustness and high inference efficiency compared with the least square based methods. Under some mild conditions, we show that Wilks’ theorem of the proposed empirical likelihood approach continues to hold. Advantages of empirical likelihood modal regression as a nonparametric approach are illustrated by constructing confidence intervals/regions. Two simulation studies and a real data analysis confirm our theoretical findings.

Appendix

1.1 Proof of Theorem 1

Proof

We first prove the root-$n$ consistency of ${\hat{\varvec{\beta }}}$, i.e., $\Vert \hat{\varvec{\beta }}-{\varvec{\beta }}_0\Vert =O_p(n^{-1/2})$. It is sufficient to show that for any given $\varrho > 0$, there exists a large constant $C$ such that

$$\begin{aligned} P\left\{ \sup _{\Vert {\varvec{v}}\Vert =C} Q_h({\varvec{\beta }}_{0}+ n^{-1/2} {\varvec{v}}) < Q_h({\varvec{\beta }}_{0})\right\} \ge 1-\varrho , \end{aligned}$$

(14)

where the function $Q_h(\cdot )$ is defined in (2).

For any vector ${\varvec{v}}$ with length $C$, by the second-order Taylor expansion, we have

$$\begin{aligned}&nQ_h\left( {\varvec{\beta }}_{0}+ n^{-1/2} {\varvec{v}}\right) -n Q_h({\varvec{\beta }}_{0}) \nonumber \\&\quad = \sum _{i=1}^n\left\{ \phi _h\left( \epsilon _i- n^{-1/2} {\varvec{x}}_i^T{\varvec{v}}\right) -\phi _h(\epsilon _i)\right\} \nonumber \\&=- \sum _{i=1}^n \phi _h^{\prime }(\epsilon _i)n^{-1/2} {\varvec{x}}_i^T{\varvec{v}} +\sum _{i=1}^n \frac{1}{2} \phi _h^{\prime \prime }(\epsilon _i)\left( n^{-1/2} {\varvec{x}}_i^T{\varvec{v}}\right) ^2\nonumber \\&\quad -\sum _{i=1}^n\frac{1}{6}\phi _h^{\prime \prime \prime }(\xi _i)\left( n^{-1/2} {\varvec{x}}_i^T{\varvec{v}}\right) ^3 \nonumber \\&\equiv I_1 + I_2 + I_3, \end{aligned}$$

(15)

where $\xi _i$ lies between $\epsilon _i$ and $\epsilon _i-n^{-1/2} {\varvec{x}}_i^T{\varvec{v}}$.

We study respectively the magnitudes of $I_1, I_2$ and $I_3$. Let $A_n {=}\! \sum _{i=1}^n \phi _h^{\prime }(\epsilon _i)n^{-1/2} {\varvec{x}}_i$. It follows from condition (C1) and $\hbox { E}(\phi '_h(\epsilon )) = 0$ that,

$$\begin{aligned} \hbox {Var}\left( A_n \right) {=} \hbox {E}\left\{ \phi _h^{\prime }(\epsilon _i) \right\} ^2 \hbox {Var}( {\varvec{x}}_i)=G(h) \Sigma . \end{aligned}$$

(16)

The finiteness of $\hbox {Var}( {\varvec{x}}_i )$ and $G(h) = {\varvec{E}}(\phi '(\epsilon )^2)$ implies that

$$\begin{aligned} \max _{1\le i\le n}\left| \phi _h^{\prime }(\epsilon _i)n^{-1/2} {\varvec{x}}_i\right| = o_p(1). \end{aligned}$$

(17)

Then by central limit theorem, we have for fixed $C$ that $A_n \overset{d}{\longrightarrow }N( 0, G(h) \Sigma )$, and therefore $I_1 \overset{d}{\longrightarrow }N( 0, G(h) {\varvec{v}}^T\Sigma {\varvec{v}})$.

For $I_2$, with the strong law of large numbers, we have $I_2 = \frac{1}{2}F(h){\varvec{v}}^T\Sigma {\varvec{v}} + o(1)$, where $F(h)$ is defined in condition (C1).

About $I_3$, we find that

$$\begin{aligned} |I_3|&\le \left| \sum _{i=1}^n\frac{1}{6}\phi _h^{\prime \prime \prime }(\xi _i)(n^{-1/2}{\varvec{x}}_i^T{\varvec{v}})^2\right| \cdot \max _{1\le i\le n} \left( |{\varvec{x}}_i^T{\varvec{v}}|/\sqrt{n}\right) \nonumber \\&\le \left| \frac{1}{6n} \sum _{i=1}^n \rho _{h,c}(\epsilon _i)( {\varvec{x}}_i^T{\varvec{v}})^2\right| \cdot \max _{1\le i\le n} \left( \Vert {\varvec{x}}_i\Vert /\sqrt{n}\right) \Vert {\varvec{v}}\Vert . \end{aligned}$$

(18)

Condition (C2) implies that $\frac{1}{6n} \sum _{i=1}^n \rho _{h,c}(\epsilon _i)( {\varvec{x}}_i^T{\varvec{v}})^2 = O_p(1).$ It then follows from the fact that $\max _{1\le i\le n} (\Vert {\varvec{x}}_i\Vert /\sqrt{n}) = o_p(1)$ that

$$\begin{aligned} I_3 = O_p(1) \cdot o_p(1) \cdot O_p(1) = o_p(1). \end{aligned}$$

Overall, we obtain that for any ${\varvec{v}}$ with $\Vert {\varvec{v}}\Vert = C$,

$$\begin{aligned} nQ_h({\varvec{\beta }}_{0}+ n^{-1/2} {\varvec{v}})-n Q_h({\varvec{\beta }}_{0}) = - A_n^T{\varvec{v}} + (1/2)F(h){\varvec{v}}^T\Sigma {\varvec{v}} + \delta _n \end{aligned}$$

with $\delta _n = o_p(1)$. The fact $- A_n^T{\varvec{v}} \overset{d}{\longrightarrow }N( 0, G(h) {\varvec{v}}^T\Sigma {\varvec{v}})$ implies that for any $\varrho >0$ and any nonzero ${\varvec{v}}$, there exists $K>0$ such that

$$\begin{aligned} P\left( \left| A_n^T{\varvec{v}}\right| <K \sqrt{G(h) {\varvec{v}}^T\Sigma {\varvec{v}}}\right) >1-\varrho . \end{aligned}$$

Thus with probability $1-\varrho$, it holds that

$$\begin{aligned} nQ_h\left( {\varvec{\beta }}_{0}+ n^{-1/2} {\varvec{v}}\right) -n Q_h({\varvec{\beta }}_{0}) \le K \sqrt{G(h) {\varvec{v}}^T\Sigma {\varvec{v}}} + (1/2)F(h){\varvec{v}}^T\Sigma {\varvec{v}} + \delta _n. \end{aligned}$$

Note that $F(h)<0$. Clearly, when $n$ and $C$ are both large enough,

$$\begin{aligned} K \sqrt{G(h) {\varvec{v}}^T\Sigma {\varvec{v}}} + (1/2)F(h){\varvec{v}}^T\Sigma {\varvec{v}} + \delta _n<0. \end{aligned}$$

In summary, for any $\varrho >0$, there exists $C>0$ such that for ${\varvec{v}}=C$, $nQ_h({\varvec{\beta }}_{0}+ n^{-1/2} {\varvec{v}})-n Q_h({\varvec{\beta }}_{0})$ is negative with probability at least $1-\varrho$. Thus, (14) holds. That is, with the probability approaching 1, there exists a local maximizer ${hat{\varvec{\beta }}}$ such that $\Vert \hat{\varvec{\beta }}-{\varvec{\beta }}_0\Vert =O_p(1/\sqrt{n})$.

We turn to proving the asymptotical normality of $\hat{\varvec{\beta }}$. Denote $\hat{\varvec{\gamma }}=\hat{\varvec{\beta }}-{\varvec{\beta }}_0$, then $\hat{\varvec{\gamma }}$ satisfies the following equation

$$\begin{aligned} 0&= \frac{1}{n}\sum _{i=1}^n{\varvec{x}}_i\phi _h^{\prime }(\epsilon _i-{\varvec{x}}_i^T\hat{\varvec{\gamma }}) \nonumber \\&= \frac{1}{n} \sum _{i=1}^n{\varvec{x}}_i\left\{ \phi _h^{\prime }(\epsilon _i)-\phi _h^{\prime \prime }(\epsilon _i){\varvec{x}}_i^T\hat{\varvec{\gamma }}+\frac{1}{2}\phi _h^{\prime \prime \prime }(\epsilon _i^*)\left( {\varvec{x}}_i^T\hat{\varvec{\gamma }}\right) ^2\right\} \nonumber \\&\triangleq J_1+J_2 \hat{\gamma }+J_3, \end{aligned}$$

(19)

where $\epsilon _i^*$ lies between $\epsilon _i$ and $\epsilon _i-{\varvec{x}}_i^T\hat{\varvec{\gamma }}$. We have shown that

$$\begin{aligned} \sqrt{n} J_1 \mathop {\longrightarrow }\limits ^{\hbox { d}}N(0, G(h){\varvec{\Sigma }}), \quad J_2 \mathop {\longrightarrow }\limits ^{\hbox { p}} F(h){\varvec{\Sigma }}. \end{aligned}$$

Meanwhile the fact $\hat{\varvec{\gamma }}= O_p(n^{-1/2})$ and condition (C2) implies that $J_3 = o_p(1)$. Thus Eq. (19) implies ${\hat{\varvec{\gamma }}} = - J_2^{-1} J_1 + o_p(1).$ Since the bandwidth $h$ is a constant not depending on $n$, by Slutsky’s theorem, we have

$$\begin{aligned} \sqrt{n}( \hat{\varvec{\beta }}-{\varvec{\beta }}_0) = \sqrt{n}{\hat{\varvec{\gamma }}} \mathop {\longrightarrow }\limits ^{\hbox { d}}N(0, {\varvec{\Sigma }}^{-1})\{ G(h)/F^2(h)\}. \end{aligned}$$

$\square$

The following lemma is needed to prove Theorem 2.

Lemma 1

Under the conditions of Theorem 1, the ${\mathbf \lambda }_{\beta _0}$ in (10) satisfies $\Vert {\varvec{\lambda }}_{\beta _0}\Vert =O_p(n^{-1/2})$.

Proof

Denote ${\varvec{\lambda }}_{\beta _0}=\zeta {\mathbf u}_0$ with ${\mathbf u}_0$ a unit vector and $\zeta =\Vert {\varvec{\lambda }}_{\beta _0}\Vert$. Define matrix ${\varvec{\Phi }}_n({\varvec{\beta }})=n^{-1} \sum _{i=1}^n \xi _i({\varvec{\beta }})\xi _i^T({\varvec{\beta }})$ and $Z=\max _{1\le i\le n}\Vert \xi _i({\varvec{\beta }}_0)\Vert$. It follows from the definition of ${\varvec{\lambda }}_{\beta _0}$ that

$$\begin{aligned} 0&= \frac{{\mathbf u}_0^T}{n}\sum _{i=1}^n \frac{\xi _i ({\varvec{\beta }}_0)}{1+\zeta {\mathbf u}_0^T\xi _i({\varvec{\beta }}_0)} =\frac{{\mathbf u}_0^T}{n}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0)- \frac{\zeta }{n}\sum _{i=1}^n \frac{\{{\mathbf u}_0^T\xi _i ({\varvec{\beta }}_0)\}^2}{1+\zeta {\mathbf u}_0^T\xi _i({\varvec{\beta }}_0)} \\&\le \frac{{\mathbf u}_0^T}{n}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0)-\frac{\zeta }{1+\zeta Z} \frac{1}{n}\sum _{i=1}^n ({\mathbf u}_0^T\xi _i ({\varvec{\beta }}_0))^2 \\&= \frac{{\mathbf u}_0^T}{n}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0)-\frac{\zeta }{1+\zeta Z}{\mathbf u}_0^T{\varvec{\Phi }}_n({\varvec{\beta }}_0) {\mathbf u}_0, \end{aligned}$$

which implies

$$\begin{aligned} \zeta \left\{ {\mathbf u}_0^T{\varvec{\Phi }}_n({\varvec{\beta }}_0) {\mathbf u}_0 -Z \frac{{\mathbf u}_0^T}{n}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0) \right\} \le \frac{{\mathbf u}_0^T}{n}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0). \end{aligned}$$

(20)

By the Cauchy–Schwarz inequality and law of large numbers, we have

$$\begin{aligned} \left| \frac{{\mathbf u}_0^T}{n}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0)\right| \le \left\| \frac{1}{n} \sum _{i=1}^n\xi _i({\varvec{\beta }}_0)\right\| =O_p(n^{-1/2}). \end{aligned}$$

(21)

This together with Eq. (17) gives

$$\begin{aligned} Z \frac{{\mathbf u}_0^T}{n}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0) =o_p(1). \end{aligned}$$

(22)

Condition (C1) and law of large numbers implies ${\varvec{\Phi }}_n \mathop {\longrightarrow }\limits ^{\hbox { p}} G(h){\varvec{\Sigma }}$, which means that there exists $c>0$ such that $P({\varvec{u}}_0^T{\varvec{\Phi }}_n{\varvec{u}}_0>c)\rightarrow 1$ as $n\rightarrow \infty$.

Furthermore, since ${n}^{-1/2}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0) \mathop {\longrightarrow }\limits ^{\hbox { d}} N(0, {\varvec{\Phi }})$, we find that $\Vert {\varvec{\lambda }}_{\beta _0}\Vert =O_p(n^{-1/2})$. $\square$

1.2 Proof of Theorem 2

Proof

Let $y_i = {\varvec{\lambda }}^T_{\beta _0}\xi _i({\varvec{\beta }}_0)$. It follows from Lemma 1 that

$$\begin{aligned} \max _{1\le i\le n}|y_i| \le \Vert {\varvec{\lambda }}_{\beta _0}\Vert \max _{1\le i\le n}|\xi _i({\varvec{\beta }}_0)| =O_p(n^{-1/2})o_p(n^{1/2})=o_p(1), \end{aligned}$$

which implies that the upcoming Taylor expansion is valid. Applying the second-order Taylor expansion on $(1+y_i)^{-1}$ for $i$ from 1 to $n$, we obtain from Eq. (10) that

$$\begin{aligned} {\varvec{\lambda }}_{\beta _0}=\{{\varvec{\Phi }}_n({\varvec{\beta }}_0)\}^{-1} \frac{1}{n}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0) +\{{\varvec{\Phi }}_n({\varvec{\beta }}_0)\}^{-1}r_n({\varvec{\beta }}_0), \end{aligned}$$

(23)

where $r_n({\varvec{\beta }}_0)\ =(1/n)\sum _{i=1}^n\xi _i({\varvec{\beta }}_0) (1+\delta ^*_i)^{-1}\{{\varvec{\lambda }}^T_{\beta _0}\xi _i({\varvec{\beta }}_0)\}^2$ and $\delta ^*_i$ lies between $0$ and $y_i$. Clearly $\max _{1\le i\le n}|\delta _i^*| = o_p(1)$. Therefore

$$\begin{aligned} |r_n({\varvec{\beta }}_0)|&\le \max _{1\le i\le n}\Vert \xi _i({\varvec{\beta }}_0) \Vert (1- \max _{1\le i\le n}|\delta _i^*|)^{-1} {\varvec{\lambda }}^T_{\beta _0} {\varvec{\Phi }}_n({\varvec{\beta }}_0) {\varvec{\lambda }}_{\beta _0}\\&= o_p(n^{1/2})O_p(n^{-1}) =o_p(n^{-1/2}). \end{aligned}$$

Thus we have

$$\begin{aligned} {\varvec{\lambda }}_{\beta _0} =\{{\varvec{\Phi }}_n({\varvec{\beta }}_0)\}^{-1} \frac{1}{n}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0) +o_p(n^{-1/2}). \end{aligned}$$

(24)

Similarly, by the third-order Taylor expansion on $\log (1+y_i)$ for all $i$, we have

$$\begin{aligned} \begin{array}{lll} -2 l({\varvec{\beta }}_0) &{}=&{} 2 \sum \limits _{i=1}^n {\varvec{\lambda }}_{\beta _0}^T\xi _i({\varvec{\beta }}_0)-\sum \limits _{i=1}^n \left\{ {\varvec{\lambda }}_{\beta _0}^T\xi _i({\varvec{\beta }}_0)\right\} ^2\\ &{}&{}+ \frac{2}{3} \sum \limits _{i=1}^n \left\{ {\varvec{\lambda }}_{\beta _0}^T\xi _i({\varvec{\beta }}_0)\right\} ^3(1+\eta _i^*)^{-3} \end{array} \end{aligned}$$

(25)

where $\eta _i$ lies between $0$ and $y_i$. It can be verified that

$$\begin{aligned}&\left| \sum _{i=1}^n \left\{ {\varvec{\lambda }}_{\beta _0}^T\xi _i({\varvec{\beta }}_0)\right\} ^3(1+\eta _i^*)^{-3} \right| \\&\quad \le \max _{1\le i\le n}\left| {\varvec{\lambda }}_{\beta _0}^T\xi _i({\varvec{\beta }}_0) \right| \left( 1-\max _{1\le i\le n}|\eta _i^*|\right) ^{-3} n {\varvec{\lambda }}^T_{\beta _0} {\varvec{\Phi }}_n({\varvec{\beta }}_0) {\varvec{\lambda }}_{\beta _0} \\&\quad = o_p(1) \cdot O_p(1) = o_p(1). \end{aligned}$$

Furthermore, by incorporating Eq. (24), we have

$$\begin{aligned} -2 l({\varvec{\beta }}_0) = \left\{ {n^{-1/2}}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0)\right\} ^{T} \left\{ {\varvec{\Phi }}_n({\varvec{\beta }}_0)\right\} ^{-1} \left\{ {n^{-1/2}}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0)\right\} +o_p(1). \end{aligned}$$

(26)

Since $\xi _i({\varvec{\beta }}_0)={\varvec{x}}_i\phi _h^{\prime } \epsilon _i$, it follows from conclusion of Lemma 1 that as $n\rightarrow \infty$,

$$\begin{aligned} \left\{ {n^{-1/2}}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0)\right\} ^{T} \left\{ {\varvec{\Phi }}_n({\varvec{\beta }}_0)\right\} ^{-1} \left\{ {n^{-1/2}}\sum _{i=1}^n\xi _i({\varvec{\beta }}_0)\right\} \mathop {\longrightarrow }\limits ^{\mathrm{d}} \chi ^2_p, \end{aligned}$$

which immediately implies $-2 l({\varvec{\beta }}_0) \mathop {\longrightarrow }\limits ^{\mathrm{d}} \chi ^2_p$. This completes the proof. $\square$