An Introduction to Total Least Squares

TL;DR

The paper reframes fitting under measurement error to include errors in both the design matrix and response by presenting Total Least Squares (TLS) as a natural generalization of ordinary least squares. It develops a unified, geometry-based view using column-space and row-space interpretations, with solutions obtained via singular value decomposition (SVD) across simple and multiple regression settings. It also extends TLS to scenarios with multiple right-hand sides and with frozen columns, providing explicit SVD-based formulas and discussing existence and uniqueness conditions. Altogether, TLS offers a robust framework for accurate data approximation when errors afflict both sides of a linear model and scales to more complex, overdetermined systems.

Abstract

The method of ``Total Least Squares'' is proposed as a more natural way (than ordinary least squares) to approximate the data if both the matrix and and the right-hand side are contaminated by ``errors''. In this tutorial note, we give a elementary unified view of ordinary and total least squares problems and their solution. As the geometry underlying the problem setting greatly contributes to the understanding of the solution, we introduce least squares problems and their generalization via interpretations in both column space and (the dual) row space and we shall use both approaches to clarify the solution. After a study of the least squares approximation for simple regression we introduce the notion of approximation in the sense of ``Total Least Squares (TLS)'' for this problem and deduce its solution in a natural way. Next we consider ordinary and total least squares approximations for multiple regression problems and we study the solution of a general overdetermined system of equations in TLS-sense. In a final section we consider generalizations with multiple right-hand sides and with ``frozen'' columns. We remark that a TLS-approximation needs not exist in general; however, the line (or hyperplane) of best approximation in TLS-sense for a regression problem does exist always.

Figures (6)

• Figure 1: Vector ${\bf x}$, its orthogonal projection on $span\{{\bf e}\}$ and the residual vector ${\bf x} -z\,{\bf e}$ in the dual approach.
• Figure 2: Simple linear regression; distances are measured along the $y$-axis.
• Figure 3: Line of Total Least Squares: Model errors are distributed over the $x$- and $y$-coordinates.
• Figure 4: The line $\ell$ in the plane is given as the line through the vector ${\bf w}$ orthogonal to the vector ${\bf r}$ of unit length. For a given vector ${\bf x}$ the difference vector ${\bf x} -{\bf w}$ is drawn together with its projection along the line $\ell$ and its orthogonal complement.
• Figure 5: Components $(f_i,g_i)$ are the best approximations of $(x_i,y_i)$ on the line $a+r_1x+r_2y=0$ .