Overparameterized Multiple Linear Regression as Hyper-Curve Fitting

TL;DR

The paper tackles overparameterization in linear regression by reframing MLR as a one-dimensional hyper-curve (PARCUR) via inverse regression (IR). On Fundamentally Overparameterized (FOP) datasets of rank $q$, PARCUR/IR predictions coincide with MLR predictions, and exact prediction is achieved when the training set $S_q$ is complete with $ ext{rank}(X)=q$. It introduces polynomial regularization by truncating the monomial basis to degree $r$ and a predictor-removal scheme using column-wise residuals $oldsymbol{\chi}_j$ to enhance predictive power, with cross-validation guiding $r^*$ and the threshold $ au$. Demonstrated on synthetic data and the Yarn chemometric dataset, the approach yields substantial improvements in predictive accuracy and interpretable per-predictor relations, while suggesting extensions to higher-dimensional manifolds for complex predictor–response structures.

Abstract

The paper shows that the application of the fixed-effect multiple linear regression model to an overparameterized dataset is equivalent to fitting the data with a hyper-curve parameterized by a single scalar parameter. This equivalence allows for a predictor-focused approach, where each predictor is described by a function of the chosen parameter. It is proven that a linear model will produce exact predictions even in the presence of nonlinear dependencies that violate the model assumptions. Parameterization in terms of the dependent variable and the monomial basis in the predictor function space are applied here to both synthetic and experimental data. The hyper-curve approach is especially suited for the regularization of problems with noise in predictor variables and can be used to remove noisy and "improper" predictors from the model.

Figures (7)

• Figure 1: Examples of column data produced by non-functional relationships between the dependent variable $y$ and the predictor variables $x_{1}$ and $x_{2}$. Left: data on a curve that cannot be parameterized by $y$. Right: data on a conical surface. Two-dimensional scatter-plots: $x_{1}$ and $x_{2}$ column data sorted by $y$ and displayed as 'functions' of $y$.
• Figure 1: \newlabelfig:Regularization0 Top: application of the regularization procedure in the case of exact predictor data $X$ and noisy dependent-variable data ${\bm y}+{\bm \epsilon}$, ${\bm \epsilon}\sim{\mathcal{N}}({\bm 0},\sigma_{i}^{2}I)$, with $\sigma_{1}=0.05$ (blue), $\sigma_{2}=0.1$ (orange), and $\sigma_{3}=0.2$ (green). Bright colored lines: validation errors $\rho({\bm y}_{\rm v})$ (left, right vertical axis) and $\rho(X_{\rm v})$ (right). Dim dashed gray lines: errors on the test dataset, $\rho({\bm y}_{\rm t})$ (left, left vertical axis) and $\rho(X_{\rm t})$ (right). Bright colored square markers: values of the test errors $\rho({\bm y}_{\rm t})$ and $\rho({X}_{\rm t})$ attained at $r^{*}$ (left and right respectively). Bright colored diamond markers: values of the test errors $\rho({\bm y}_{\rm t})$ (left, left vertical axis) and $\rho({X}_{\rm t})$ (right) attained with $r^{*}$ and the optimal set $\{p\}_{\text{opt}}$ of predictors. Bottom: same as in the row above, but with a third of the columns of $X$ affected by the additive Gaussian noise of the same type as the noise in the dependent variable ${\bm y}$.
• Figure 1: \newlabelfig:FeatureRemovalError0 Feature removal errors $\rho({\bm y})$ (solid blue) and $\rho({\bm y}_{\rm t})$ (dashed gray), both with $r=r^{*}$, for synthetic datasets containing additive noise in the dependent variable and one-third of predictors, $\sigma=0.05$ (left) and $\sigma=0.1$ (right). Red squares indicate the minima, corresponding to the optimal thresholds $\tau_{\rm opt}$.
• Figure 1: (a): column-wise prediction errors $\chi_{j}$ and the optimal threshold $\tau_{\rm opt}$ (top); rows of $X$ displayed as curves (middle); first three rows of $\hat{A}$ displayed as curves (bottom). Removed predictors are shown a vertical light-gray lines (middle, bottom). (b): predictor data and the corresponding polynomial fits for the retained (left) and removed (right) predictors.
• Figure 2: Top: prediction errors of the IR model as functions of the training dataset rank obtained with invertible basis matrix $V$ and exact data (see text for full explanation). Bottom: test of the regularization algorithm on exact (noiseless) data and over-complete training dataset. Bright enlarged colored markers correspond to the minima of the validation errors that indicate the optimal polynomial degree $r^{*}$ along the horizontal axis (bottom, right) and show the values of the test errors attained with these $r^{*}$ (bottom, left). Note, the horizontal axis displays ${\rm rank}(\hat{A})=r+1$, rather than the actual polynomial degree $r$.

Theorems & Definitions (8)

• Definition 1
• Definition 2
• Theorem 3
• Proof 1
• Theorem 4
• Proof 2
• Theorem 1
• Proof 3