Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Specular differentiation in normed vector spaces and its application to nonsmooth convex optimization

Kiyuob Jung

TL;DR

This work generalizes differentiability via specular differentiation to normed spaces and develops a coherent theory around specular directional, Gâteaux, and Fréchet derivatives, culminating in the specular gradient concept. It then leverages these notions to design three convex optimization algorithms—SPEG, S-SPEG, and H-SPEG—for nonsmooth problems in $\mathbb{R}^n$, providing theoretical convergence guarantees and practical performance insights. The authors demonstrate that specular gradients act as subgradients for convex functions and establish quasi-mean value and quasi-Fermat theorems in the specular setting, along with representations using $\mathcal{A}$ and $\mathcal{B}$. Numerical experiments on Elastic Net objectives show that the proposed methods can converge where classical methods fail and that stochastic and hybrid variants improve computational efficiency. Overall, the paper provides a solid theoretical foundation and actionable algorithms for nonsmooth convex optimization using specular differentiation, supported by a publicly available Python package.

Abstract

This paper introduces specular differentiation, which generalizes Gâteaux and Fréchet differentiation in normed vector spaces. Fundamental theoretical properties of specular differentiation are investigated, including the Quasi-Mean Value Theorem and Quasi-Fermat's Theorem. As an application, three numerical methods using specular differentiation are devised to optimize nonsmooth convex functions in higher-dimensional Euclidean spaces. Numerical experiments demonstrate that the proposed methods are capable of minimizing non-differentiable functions that classical methods fail to minimize.

Specular differentiation in normed vector spaces and its application to nonsmooth convex optimization

TL;DR

This work generalizes differentiability via specular differentiation to normed spaces and develops a coherent theory around specular directional, Gâteaux, and Fréchet derivatives, culminating in the specular gradient concept. It then leverages these notions to design three convex optimization algorithms—SPEG, S-SPEG, and H-SPEG—for nonsmooth problems in , providing theoretical convergence guarantees and practical performance insights. The authors demonstrate that specular gradients act as subgradients for convex functions and establish quasi-mean value and quasi-Fermat theorems in the specular setting, along with representations using and . Numerical experiments on Elastic Net objectives show that the proposed methods can converge where classical methods fail and that stochastic and hybrid variants improve computational efficiency. Overall, the paper provides a solid theoretical foundation and actionable algorithms for nonsmooth convex optimization using specular differentiation, supported by a publicly available Python package.

Abstract

This paper introduces specular differentiation, which generalizes Gâteaux and Fréchet differentiation in normed vector spaces. Fundamental theoretical properties of specular differentiation are investigated, including the Quasi-Mean Value Theorem and Quasi-Fermat's Theorem. As an application, three numerical methods using specular differentiation are devised to optimize nonsmooth convex functions in higher-dimensional Euclidean spaces. Numerical experiments demonstrate that the proposed methods are capable of minimizing non-differentiable functions that classical methods fail to minimize.
Paper Structure (14 sections, 25 theorems, 196 equations, 6 figures, 3 tables, 2 algorithms)

This paper contains 14 sections, 25 theorems, 196 equations, 6 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Definitions and notations
  3. Numerical application
  4. Main results
  5. Organization
  6. Specular differentiation
  7. Specular directional derivatives
  8. Specular Gâteaux derivatives
  9. Specular Fréchet differentials
  10. Specular gradients
  11. Nonsmooth convex optimization in Euclidean spaces
  12. Nonsmooth convex analysis
  13. Numerical results
  14. Convergence results

Key Result

Proposition 2.1

Let $X$ and $Y$ be normed vector spaces over $\mathbb{R}$ equipped with norms $\left\| \, \cdot \, \right\|_X$ and $\left\| \, \cdot \, \right\|_Y$, respectively. Let $f : \Omega \to Y$ be a function, where $\Omega$ is an open subset of $X$. Let $x \in \Omega$ and $v \in X$ be fixed. If the directio

Figures (6)

  • Figure 1: Relations between classical and specular differentiability.
  • Figure 1: Motivation of the definition of specular directional derivatives.
  • Figure 1: Optimization of the objective function \ref{['ex:obj']} with $m=50$, $n=100$, $\lambda_1 = 0.01$, and $\lambda_2 = 1.0$.
  • Figure 2: Comparison of SPEG, S-SPEG, and H-SPEG for optimization of the objective function \ref{['ex:obj']} with $m=100$ and $n=100$ under varying $\lambda_1$ and $\lambda_2$.
  • Figure 3: Optimization for the objective function \ref{['ex:obj']} with $m=500$, $n=100$, $\lambda_1 = 100.0$, and $\lambda_2 = 1.0$.
  • ...and 1 more figures

Theorems & Definitions (62)

  • Definition 1.1
  • Definition 1.2
  • Definition 1.3
  • Proposition 2.1
  • Proof 1
  • Example 1
  • Lemma 2.2
  • Proof 2
  • Remark 2.3
  • Theorem 2.4
  • ...and 52 more