Numerical Linear Algebra: Least Squares, QR and SVD
Davoud Mirzaei
TL;DR
The notes present a comprehensive treatment of numerical linear algebra for overdetermined systems, focusing on least squares, orthogonal factorizations, and the SVD. They compare normal equations with stable alternatives such as QR and SVD, emphasize conditioning and the role of the pseudoinverse, and provide detailed algorithmic guidance for Householder and Givens QR factorizations, including column pivoting for rank-deficient problems. The SVD is developed as a unifying framework for LS, pseudoinversion, and optimal low-rank approximations, with broad applications to data analysis, image processing, PCA, and pattern recognition. Together, the sections outline both theory and practical algorithms (with complexity and stability considerations) that underpin robust numerical linear algebra in scientific computing.
Abstract
These lecture notes focus on some numerical linear algebra algorithms in scientific computing. We assume that students are familiar with elementary linear algebra concepts such as vector spaces, systems of equations, matrices, norms, eigenvalues, and eigenvectors. In the numerical part, we do not pursue Gaussian elimination and other LU factorization algorithms for square systems. Instead, we mainly focus on overdetermined systems, least squares solutions, orthogonal factorizations, and some applications to data analysis and other areas.