A Cautionary Note on the Use of Linear Regression for Hypothesis Testing

This note draws researchers’ attention to the use of linear regression for the purpose of conducting a hypothesis testing. Even when multiple explanatory variables are included in a regression equation to preclude the hazard of a simple regression of omitting other factors, multicollinearity unfortunately is inherent in multiple regression simply because the included explanatory variables can share some common parameter domain; that is, they co-vary. Here we shall show that however one transforms the variance-covariance matrix of the least-squares estimation to reduce the estimation errors, the procedure amounts to affine transformations of the explanatory variables so that despite the transformation they remain to co-vary, rendering the coefficient as a partial derivative invalid. The root of this problem originates from the fact that one receives the explanatory variables’ values by observation rather than predetermining their values and then collecting the corresponding dependent variable’ values. This situation becomes especially disconcerting when a transformed explanatory variable has its estimated coefficient enjoying an exceptional degree of confidence but of no mathematical status as a partial derivative, misleading engineering or medical prescriptions and public policies.

Introduction

It is a well-recognized fact that a simple linear regression can cause the problem of omitted variables or model misspecification. Thus, multiple regression serves as a better methodology; here, however, one must exercise caution about the choices of the explanatory variables and also the design of the sample. In empirical work, there are mainly two approaches: observational (as in astronomy) or experimental (as in a particle collider). In practical applications of regression, mostly it is the former: one draws a sample from the underlying population and observes the input data [1]. This presents problems, as the analyst has no control over the observed values, say, $\left\{(x_{i1},x_{i2};y_i)|i=1,2,\cdots ,n\right\}$. This note seeks to draw researchers’ attention to a highly likely situation where $x_1$ and $x_2$ are both functions of $t$ and $s$ (connoting time and space), i.e., $x_1(t,s)$ and $x_2(t,s)$, so that the very construct of $E\left(Y\right)=\alpha +{\beta }_1{X}_1+{\beta }_2{X}_2$ is immediately invalid as ${\beta }_j=(\mathrm{\partial }E\left(Y\right))/(\mathrm{\partial }{X}_j),j=1,2$, does not exist. In the next section we will show that all modifications of the “variance-covariance matrix” ($X^{T}X$) of the least-squares estimation for the purpose of alleviating the problem of multicollinearity cannot treat this problem of explanatory variables sharing the same parameter domain. Then in Section 3 we will conclude with a summary remark.

Analysis

All remedial procedures against the problem of multicollinearity are based on modifications of the matrix $\left(X^{T}X\right)$ of the ordinary least-squares estimation [2], in the case of two explanatory variables,

(1) $$\begin{array}{r}{X}^{T}X=\left(\begin{array}{ll}\sum _{i=1}^n{(x_{i1}-{\overline{x}}_1)}^2& \sum _{i=1}^n(x_{i1}-{\overline{x}}_1)(x_{i2}-{\overline{x}}_2)\\ \sum _{i=1}^n(x_{i1}-{\overline{x}}_1)(x_{i2}-{\overline{x}}_2)& \sum _{i=1}^n{(x_{i2}-{\overline{x}}_2)}^2\end{array}\right).\end{array}$$

Since $\left(X^{T}X\right)$ is a linear operator, transformations of which are in general linear or affine [3], [4]. Consider a linear transformation by $k_1>1$ of the $(1,1)-entry$ of the above matrix, then each individual observation $i$ of $x_1(t,s)$ undergoes an affine transformation, which still ends as a function of $(t,s)$. Consider now an affine transformation of the $(1,1)-entry$ of the above matrix by $M_1>0$, i.e., but then the above (4) can be re-expressed as (2),

(2) $$\begin{array}{r}{k}_1^2\sum _i{(x_{i1}-{\overline{x}}_1)}^2=\sum _i{(k_1{x}_{i1}-k_1{\overline{x}}_1)}^2;\end{array}$$

(3) $$\begin{array}{r}{k}_1\cdot x_{i1}(t_i,s_i)-\left(k_1{\overline{x}}_1\right),\end{array}$$

(4) $$\begin{array}{r}{M}_1+\sum _i{(x_{i1}-{\overline{x}}_1)}^2,\end{array}$$

(5) $$\begin{array}{l}{M}_1+\sum _i{(x_{i1}-{\overline{x}}_1)}^2=k_1^2\sum _i{(x_{i1}-{\overline{x}}_1)}^2=\sum _i{(k_1{x}_{i1}-k_1{\overline{x}}_1)}^2;\\ \text{with }{k}_1^2=\frac{M_1}{\sum {(x_{i1}-{\overline{x}}_1)}^2}+1.\end{array}$$

Thus, each diagonal entry of $\left(X^{T}X\right)$ remains as a function of $(t,s)$ following any linear/affine transformations, and thereof each transformed explanatory variable ${\text{a}}^{{}^{\prime }}\text{ la}$ (3) is still a function of $(t,s)$. Next, consider the off-diagonal entries of $\left(X^{T}X\right)$: Denote the transformed $x_1$ and $x_2$ by $x_1^{\ast }$ and $x_2^{\ast }$ respectively; then one has

also still a function of $(t,s)$.

(6) $$\begin{array}{l}\sum _i(x_{i1}^{\ast }-{\overline{x}}_1^{\ast })(x_{i2}^{\ast }-{\overline{x}}_2^{\ast })=⟨k_1{x}_{1,n\times 1}-k_1{\overline{x}}_{1,n\times 1},k_2{x}_{2,n\times 1}-k_2{\overline{x}}_{2,n\times 1}⟩\\ =k_1{k}_2\cdot (⟨x_1,x_2⟩-⟨x_1,{\overline{x}}_2⟩-⟨{\overline{x}}_1,x_2⟩+⟨{\overline{x}}_1,{\overline{x}}_2⟩),\end{array}$$

As such, any linear or affine transformation of the matrix $\left(X^{T}X\right)$ amounts to a transformation of an explanatory variable $x_j$ into $k_j\cdot x_j(t,s)-m_j=f_j(t,s)$. Consequently, the transformed regression equation remains as

i.e., the root problem of multicollinearity remains: while the t-statistics of $b_1,b_2,\cdots ,b_k$ can be increased to high (absolute) values to the extent that the $99\mathrm{\%}$ confidence intervals around $b_1,b_2,\cdots ,b_k$ are extremely narrow, one still has the problem

(7) $$\begin{array}{r}{\hat{y}}^{\ast }=b_1{x}_1^{\ast }(t,s)+b_2{x}_2^{\ast }(t,s)+\cdots +b_k{x}_k^{\ast }(t,s);\end{array}$$

by the simple fact that $x_j^{\ast }=f_j(t,s),\mathrm{\forall }j=1,2,\cdots ,k$; i.e., all the explanatory variables co-vary. Therefore, the engineered highly significant estimates of the coefficients can only lead to a false confidence in their values [5]–[7].

(8) $$\begin{array}{r}E\left(b_j\right)={\beta }_j=\frac{\mathrm{\partial }E\left(Y\right)}{\mathrm{\partial }{X}_j^{\ast }},j=1,2,\cdots ,k,\text{do not exist},\end{array}$$

Summary Remark

From the above analysis, we see that hypothesis testing by a regression equation without an underlying mathematical model as based on theories can be problematic, unless one fixes the explanatory variables’ input values $\left\{(x_{i1},x_{i2},\cdots ,x_{ik})|i=1,2,\cdots ,n\right\}$ in advance and then collects the dependent variable’s values $\left\{y_i|i=1,2,\cdots ,n\right\}$. Otherwise, no matter how one transforms the matrix $\left(X^{T}X\right)$, the problem of $x_j^{\ast }(t,s)\mathrm{\forall }j$ persists; then the transformation not only is futile but can also be dangerous—considering a wrong sign of a coefficient (in the sense of $\mathrm{\partial }{X}_j/\mathrm{\partial }t>0,\mathrm{\partial }{X}_j/\mathrm{\partial }s>0,\mathrm{\partial }Y/\mathrm{\partial }t>0,\mathrm{\partial }Y/\mathrm{\partial }s>0\text{ but}{\text{b}}_j<0$) that yet carries a high degree of confidence in engineering or medical applications.