Preconditioned Conjugate Gradient methods for the estimation of General Linear Models

TL;DR

This work presents PCG-Aug, a preconditioned conjugate gradient method that solves GLS estimators for generalized linear models by recasting GLS as an augmented system. The method employs an indefinite preconditioner to blend efficient OLS steps with iterative updates, enabling exact GLS results in a finite number of iterations while accommodating singular or near-singular covariance structures. The authors derive statistical properties showing that intermediate estimators are unbiased and demonstrate robustness and speed on constrained GLMs, SUR, and VAR-like models, where traditional direct solvers may be expensive or unstable. Empirical results show PCG-Aug matching the precision of direct methods at a fraction of the computational cost, with particular strength in structured problems where exploiting block or Kronecker structures yields substantial gains.

Abstract

The use of the Preconditioned Conjugate Gradient (PCG) method for computing the Generalized Least Squares (GLS) estimator of the General Linear Model (GLM) is considered. The GLS estimator is expressed in terms of the solution of an augmented system. That system is solved by means of the PCG method using an indefinite preconditioner. The resulting method iterates a sequence Ordinary Least Squares (OLS) estimations that converges, in exact precision, to the GLS estimator within a finite number of steps. The numerical and statistical properties of the estimator computed at an intermediate step are analytically and numerically studied. This approach allows to combine direct methods, used in the OLS step, with those of iterative methods. This advantage is exploited to design PCG methods for the estimation of Constrained GLMs and of some structured multivariate GLMs. The structure of the matrices involved are exploited as much as possible, in the OLS step. The iterative method then solves for the unexploited structure. Numerical experiments shows that the proposed methods can achieve, for these structured problems, the same precision of state of the art direct methods, but in a fraction of the time.

Figures (8)

• Figure 1: Eigenvalue distribution of $\bf{KG}$ (left panel) and convergence profile (right panel). The convergence is expressed in terms of the root-mean-square error $\text{RMSE} = n^{-\frac{1}{2}}\|\hat{\bm{z}}_i - \bm\beta\|_2$.
• Figure 2: Convergence profiles for $\alpha = 2$ (left) and $\alpha = 8$ (right). The RMS (root-mean-square) of the error $\hat{\bm{z}}_i - \bm\beta$ and of the residuals $\bm{y} - \bm{X}\hat{\bm{z}}_i - \bm\Sigma\hat{\bm\omega}_i$ and $\bm{X}^T\hat{\bm\omega}_i$ are shown.
• Figure 3: Spectrum of $\bm{KG}$.
• Figure 4: Montecarlo average and 95% and 99% bands for the error on the second element of the PCG-Aug and PCG-NE estimators for $\bm\beta$. Analogous statistics for the GLS estimator are shown as horizontal continous, dashed and dotted lines.
• Figure 5: Montecarlo average and 95% and 99% bands for the RMSE error $n^{-\frac{1}{2}} \| \hat{\bm{z}}_i - \bm\beta \|_{(\bm{X}^T\bm\Sigma^{-1}\bm{X})}$ of the PCG-Aug and PCG-NE estimators.

Theorems & Definitions (10)

• Lemma 1
• Lemma 2
• Corollary 1
• Theorem 1
• Lemma 3
• proof
• Lemma 4
• proof
• Remark 1
• proof : Proof of Lemma \ref{['thm:AugBLUE']}