Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Twice Epi-Differentiability of Spectral Functions and its applications

Chao Ding, Ebrahim Sarabi, Shiwei Wang

TL;DR

The paper develops a unified second-order variational framework for spectral functions by showing that twice epi-differentiability of a spectral function g(X)=(\theta\circ\lambda)(X) is equivalent to the same property for its symmetric part θ, even without convexity. It derives an explicit formula for the second subderivative d^2g(X,Y)(H) in terms of the symmetric part and eigenstructure, and uses this to characterize proto-differentiability of subgradients and the directional differentiability of proximal mappings. These results are then applied to leading eigenvalue functions and practical spectral regularizers (e.g., MCP penalties, largest eigenvalue gaps) to obtain concrete second-order properties and algorithmic implications. The findings enable robust analysis and efficient optimization strategies for a broad class of nonconvex spectral problems in statistics and machine learning, including robust PCA.

Abstract

Second-order variational properties have been shown to play important theoretical and numerical roles for different classes of optimization problems. Among such properties, twice epi-differentiability has a special place because of its ubiquitous presence in various classes of extended-real-valued functions that are important for optimization problems. We provide a useful characterization of this property for spectral functions by demonstrating that it can be characterized via the same property of the symmetric part of the spectral representation of an eigenvalue function. Our approach allows us to bypass the rather restrictive convexity assumption, used in many recent works that targeted second-order variational properties of spectral functions. By this theoretical tool, several applications on the proto-differentiability of subgradient mappings, the directional differentiability of the proximal mapping of spectral functions are achieved. We finally use our established theory to study twice epi-differentiability of leading eigenvalue functions and practical regularization terms that have important applications in statistics and the robust PCA.

Twice Epi-Differentiability of Spectral Functions and its applications

TL;DR

The paper develops a unified second-order variational framework for spectral functions by showing that twice epi-differentiability of a spectral function g(X)=(\theta\circ\lambda)(X) is equivalent to the same property for its symmetric part θ, even without convexity. It derives an explicit formula for the second subderivative d^2g(X,Y)(H) in terms of the symmetric part and eigenstructure, and uses this to characterize proto-differentiability of subgradients and the directional differentiability of proximal mappings. These results are then applied to leading eigenvalue functions and practical spectral regularizers (e.g., MCP penalties, largest eigenvalue gaps) to obtain concrete second-order properties and algorithmic implications. The findings enable robust analysis and efficient optimization strategies for a broad class of nonconvex spectral problems in statistics and machine learning, including robust PCA.

Abstract

Second-order variational properties have been shown to play important theoretical and numerical roles for different classes of optimization problems. Among such properties, twice epi-differentiability has a special place because of its ubiquitous presence in various classes of extended-real-valued functions that are important for optimization problems. We provide a useful characterization of this property for spectral functions by demonstrating that it can be characterized via the same property of the symmetric part of the spectral representation of an eigenvalue function. Our approach allows us to bypass the rather restrictive convexity assumption, used in many recent works that targeted second-order variational properties of spectral functions. By this theoretical tool, several applications on the proto-differentiability of subgradient mappings, the directional differentiability of the proximal mapping of spectral functions are achieved. We finally use our established theory to study twice epi-differentiability of leading eigenvalue functions and practical regularization terms that have important applications in statistics and the robust PCA.
Paper Structure (8 sections, 18 theorems, 117 equations)

This paper contains 8 sections, 18 theorems, 117 equations.

Table of Contents

  1. Introduction
  2. Notations and Preliminaries
  3. Twice Epi-Differentiability of Spectral Functions
  4. Applications of Twice Epi-Differentiability
  5. Illustrated Examples
  6. Second-Order Variational Properties of Leading Eigenvalues
  7. Second-Order Variational Properties of Statistic Regularizer
  8. Conclusion

Key Result

Proposition 2.1

Assume that $X \in {\bf {S}}^n$ has the eigenvalue decomposition specdocom for some $U \in {\bf O}^n (X)$. Let $\mu_{1} > \cdots> \mu_{r}$ be distinct eigenvalues of $X$. Then, for any $H\in {\bf {S}}^n$ that $H\to 0$ and any $i\in \{1,\ldots,n\}$, we have and where $m\in \{1,\ldots,r\}$ with $i \in \alpha_m$. Thus, we have

Theorems & Definitions (21)

  • Proposition 2.1: First-order expansion of eigenvalue functions
  • Proposition 2.2
  • Theorem 2.3: Subderivatives of spectral functions
  • Proposition 3.1: Lower estimate for second subderivatives
  • Proposition 3.2: critical cone of spectral functions
  • Lemma 3.3
  • Theorem 3.4
  • Remark 3.5
  • Proposition 3.6
  • Theorem 4.1
  • ...and 11 more