Higher-Order Singular-Value Derivatives of Rectangular Real Matrices

TL;DR

The paper addresses the challenge of computing higher-order Fréchet derivatives of singular values for real rectangular matrices by embedding the matrix into a self-adjoint Jordan–Wielandt operator and applying Kato perturbation theory. It develops a refined, constructive eigenvalue expansion that yields explicit coefficients for arbitrary-order spectral variations and relates them to Fréchet derivatives of singular values, with a Kronecker-product form for practical computation. Key contributions include a general framework for arbitrary-order derivatives, a novel Kronecker-form Hessian for $n=2$, and rigorous validation against auto-differentiation that confirms accuracy to numerical precision. The approach provides a principled toolkit for spectral sensitivity analyses in random-matrix and stochastic-dynamical contexts, with potential impact on adversarial perturbation studies in deep learning and related fields.

Abstract

We present a theoretical framework for deriving the general $n$-th order Fréchet derivatives of singular values in real rectangular matrices, by leveraging reduced resolvent operators from Kato's analytic perturbation theory for self-adjoint operators. Deriving closed-form expressions for higher-order derivatives of singular values is notoriously challenging through standard matrix-analysis techniques. To overcome this, we treat a real rectangular matrix as a compact operator on a finite-dimensional Hilbert space, and embed the rectangular matrix into a block self-adjoint operator so that non-symmetric perturbations are captured. Applying Kato's asymptotic eigenvalue expansion to this construction, we obtain a general, closed-form expression for the infinitesimal $n$-th order spectral variations. Specializing to $n=2$ and deploying on a Kronecker-product representation with matrix convention yield the Hessian of a singular value, not found in literature. By bridging abstract operator-theoretic perturbation theory with matrices, our framework equips researchers with a practical toolkit for higher-order spectral sensitivity studies in random matrix applications (e.g., adversarial perturbation in deep learning).

Figures (4)

• Figure 1: Theoretical Framework for Infinitesimal Spectral Variations. We extend Kato's analytic perturbation theory for self-adjoint operators to derive arbitrary-order singular-value derivatives kato1995perturbation. For a rectangular matrix $A$, we introduce its Jordan--Wielandt embedding $\mathcal{T}$ (Theorem \ref{['theorem:Jordan_Wielandt_relation']}), a block self-adjoint operator that encodes perturbations across all subspaces (i.e., left-singular, right-singular, left-null, and right-null). By extending Kato’s asymptotic eigenvalue expansions to this embedding and expressing them in explicit closed form --- computing and simplifying with residue theorem --- yields the $n$th-order expansions of singular values of $A$. These expansions are then related to Fréchet derivatives, given by analytic perturbation theorem (Theorem \ref{['thm:frechet_taylor_operator']}). Finally, by specializing to explicit matrix-layout conventions, we obtain a systematic and constructive procedure for computing arbitrary-order singular-value derivatives of rectangular matrices. Our method is highly procedure for deriving arbitrary-order singular-value derivatives.
• Figure 2: Numerical Experiments for Singular-Value Jacobian. This experiment compares the singular-value Jacobian derived from our framework with that obtained via PyTorch's auto--differentiation. The error $\epsilon$ is measured as the $\ell_2$-norm between the theoretical and ground-truth results. The error is measured to be zero in these experiments, indicating no difference between the theoretical and ground-truth results.
• Figure 3: Numerical Experiments for Singular-Value Hessian. This experiment compares the singular-value Hessian derived from our framework with that obtained via PyTorch's auto--differentiation. The error $\epsilon$ is measured as the $\ell_2$-norm between the theoretical and ground-truth results. The maximum error is measured to be less than $1.3\times 10^{-14}$ in these experiments, indicating the difference between the theoretical and ground-truth results is negligible.
• Figure 4: Errors for Singular-Value Hessian. Random matrix entries are sampled i.i.d. from $\mathcal{N}(0,1)$ and $U[0,1]$, respectively. For each singular-value index $k=1,2,\dots,r$, the error $\epsilon$ is computed over $500$ trials and visualized using an unnormalized histogram density. All reported errors are below $6\times 10^{-14}$ in these experiments.

Theorems & Definitions (21)

• Theorem 1.1: Matrix Singular Value Decomposition (Full Form)
• Theorem 1.2: Spectrum of Jordan–Wielandt Embedding
• Remark 1.3
• Lemma 2.1: Essential Matrix Identities
• Definition 2.3: $\alpha$-Times Continuously Fréchet Differentiable Matrix Map
• Theorem 2.4: Uniqueness of $\alpha$-Times Fréchet Derivative Rudin1991spivak2018calculus
• Corollary 2.5: Vectorized Kronecker-Product Representation of Fréchet Derivative
• Definition 3.1: Space of Bounded Linear Operators
• Theorem 3.2: Kato's Weighted Mean of Eigenvalue Expansions kato1995perturbation
• Theorem 3.3: Refined Closed-Form Asymptotic Expansion of Simple Isolated Eigenvalue in Self-Adjoint Operator