Comparing the Moore-Penrose Pseudoinverse and Gradient Descent for Solving Linear Regression Problems: A Performance Analysis

TL;DR

This paper interrogates the trade-offs between the Moore–Penrose pseudoinverse and batch gradient descent for solving ordinary least squares linear regression. By combining theoretical analysis with synthetic and real-world experiments, it demonstrates that the direct pseudoinverse is typically faster and more numerically stable across moderate $n$ and $d$, while gradient descent is more sensitive to data conditioning and scales with dataset size. The study provides practical guidelines: use the pseudoinverse for moderate-sized problems with potential ill-conditioning; resort to iterative methods (and variants like SGD) for extremely large datasets, with careful preprocessing and possible hybrid strategies. Overall, the work clarifies when exact, closed-form solvers outperform iterative approaches and when scalable optimization becomes necessary, informing practitioners’ solver choices in real-world linear regression tasks.

Abstract

This paper investigates the comparative performance of two fundamental approaches to solving linear regression problems: the closed-form Moore-Penrose pseudoinverse and the iterative gradient descent method. Linear regression is a cornerstone of predictive modeling, and the choice of solver can significantly impact efficiency and accuracy. I review and discuss the theoretical underpinnings of both methods, analyze their computational complexity, and evaluate their empirical behavior on synthetic datasets with controlled characteristics, as well as on established real-world datasets. My results delineate the conditions under which each method excels in terms of computational time, numerical stability, and predictive accuracy. This work aims to provide practical guidance for researchers and practitioners in machine learning when selecting between direct, exact solutions and iterative, approximate solutions for linear regression tasks.

Figures (10)

• Figure 1: Conceptual illustration of linear regression, showing data points (blue dots) and the fitted regression line (red line) that minimizes the sum of squared residuals (vertical dashed lines).
• Figure 2: Conceptual illustration of gradient descent. The contours represent the level sets of the loss function $S(\boldsymbol{\beta})$, and the red arrows depict the iterative steps taken by the gradient descent algorithm from an initial guess (outer point) towards the minimum (center). The shape of the contours and path taken are influenced by the conditioning of the data.
• Figure 3: Visual comparison of loss surfaces for well-conditioned (left, more spherical contours) versus ill-conditioned (right, elongated contours) data. Ill-conditioning can make it significantly harder for gradient descent to find the optimal solution efficiently.
• Figure 4: Runtime (seconds) vs. Number of Features (d) for the Moore-Penrose Pseudoinverse method with well-conditioned data (cond factor=1.0) and n=1000. The plot illustrates how the direct solution's time cost scales with dimensionality.
• Figure 5: Runtime (seconds) vs. Number of Features (d) for the Gradient Descent method with well-conditioned data (cond factor=1.0) and n=1000. The plot demonstrates the scaling of the iterative solution's time cost with dimensionality.