Derivative of the truncated singular value and eigen decomposition

TL;DR

This note addresses the challenge of differentiating truncated linear algebra decompositions used in automatic differentiation. It derives explicit, implementable expressions for the differential of the truncated SVD ($t$SVD) and truncated EVD ($t$EVD), including extensions to non-square matrices and iterative solvers, as well as corrections when the full decomposition is not available. Key contributions include closed-form formulas for $dS$, $dU$, $dV$ and their eigenvector counterparts, plus Sylvester-equation-based procedures to compute missing components in the truncated setting. The results enable stable gradient-based optimization in quantum physics and related computational frameworks by providing precise, block-structured projection rules for truncated decompositions.

Abstract

Recently developed applications in the field of machine learning and computational physics rely on automatic differentiation techniques, that require stable and efficient linear algebra gradient computations. This technical note provides a comprehensive and detailed discussion of the derivative of the truncated singular and eigenvalue decomposition. It summarizes previous work and builds on them with an extensive description of how to derive the relevant terms. A main focus is correctly expressing the derivative in terms of the truncated part, despite lacking knowledge of the full decomposition.