Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lyapunov-Stable Deep Equilibrium Models

Haoyu Chu, Shikui Wei, Ting Liu, Yao Zhao, Yuto Miyatake

TL;DR

This work introduces LyaDEQ, a Lyapunov-stable Deep Equilibrium Model, to provide provable stability of fixed points defined by $F(\boldsymbol{z}^{\star})=0$. By treating DEQ dynamics as a nonlinear system and learning a convex Lyapunov function $V$ via an ICNN, the method enforces stability conditions that defend against adversarial perturbations; an orthogonal FC layer then separates Lyapunov-stable equilibria to improve class discrimination. Empirical results on MNIST, SVHN, and CIFAR-10/100 show improved robustness against white-box attacks, with further gains when combined with adversarial training methods like TRADES, RD, and PAT. The approach yields competitive clean accuracy and demonstrates substantial robustness benefits, especially at larger attack radii, while remaining compatible with existing defense strategies. Overall, the paper advances a principled, certified-stability pathway for implicit models to resist adversarial threats in practical image classification tasks.

Abstract

Deep equilibrium (DEQ) models have emerged as a promising class of implicit layer models, which abandon traditional depth by solving for the fixed points of a single nonlinear layer. Despite their success, the stability of the fixed points for these models remains poorly understood. By considering DEQ models as nonlinear dynamic systems, we propose a robust DEQ model named LyaDEQ with guaranteed provable stability via Lyapunov theory. The crux of our method is ensuring the Lyapunov stability of the DEQ model's fixed points, which enables the proposed model to resist minor initial perturbations. To avoid poor adversarial defense due to Lyapunov-stable fixed points being located near each other, we orthogonalize the layers after the Lyapunov stability module to separate different fixed points. We evaluate LyaDEQ models under well-known adversarial attacks, and experimental results demonstrate significant improvement in robustness. Furthermore, we show that the LyaDEQ model can be combined with other defense methods, such as adversarial training, to achieve even better adversarial robustness.

Lyapunov-Stable Deep Equilibrium Models

TL;DR

This work introduces LyaDEQ, a Lyapunov-stable Deep Equilibrium Model, to provide provable stability of fixed points defined by . By treating DEQ dynamics as a nonlinear system and learning a convex Lyapunov function via an ICNN, the method enforces stability conditions that defend against adversarial perturbations; an orthogonal FC layer then separates Lyapunov-stable equilibria to improve class discrimination. Empirical results on MNIST, SVHN, and CIFAR-10/100 show improved robustness against white-box attacks, with further gains when combined with adversarial training methods like TRADES, RD, and PAT. The approach yields competitive clean accuracy and demonstrates substantial robustness benefits, especially at larger attack radii, while remaining compatible with existing defense strategies. Overall, the paper advances a principled, certified-stability pathway for implicit models to resist adversarial threats in practical image classification tasks.

Abstract

Deep equilibrium (DEQ) models have emerged as a promising class of implicit layer models, which abandon traditional depth by solving for the fixed points of a single nonlinear layer. Despite their success, the stability of the fixed points for these models remains poorly understood. By considering DEQ models as nonlinear dynamic systems, we propose a robust DEQ model named LyaDEQ with guaranteed provable stability via Lyapunov theory. The crux of our method is ensuring the Lyapunov stability of the DEQ model's fixed points, which enables the proposed model to resist minor initial perturbations. To avoid poor adversarial defense due to Lyapunov-stable fixed points being located near each other, we orthogonalize the layers after the Lyapunov stability module to separate different fixed points. We evaluate LyaDEQ models under well-known adversarial attacks, and experimental results demonstrate significant improvement in robustness. Furthermore, we show that the LyaDEQ model can be combined with other defense methods, such as adversarial training, to achieve even better adversarial robustness.
Paper Structure (23 sections, 1 theorem, 9 equations, 4 figures, 3 tables)

This paper contains 23 sections, 1 theorem, 9 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Deep Equilibrium Models
  4. Lyapunov Theory in Deep Learning
  5. Preliminaries
  6. Methodology
  7. Considering a DEQ Model as a Nonlinear System
  8. Lyapunov Stability Framework
  9. LyaDEQ Model
  10. Feature extractor
  11. DEQ
  12. Lyapunov stability module
  13. Orthogonal FC layer
  14. Experiments
  15. Setup
  16. ...and 8 more sections

Key Result

Theorem 2

giesl2015review Let $\boldsymbol{u^{\star}}$ be a fixed point. Let $V: \mathcal{U} \rightarrow \mathds{R}$ be a continuously differentiable function, defined on a neighborhood $\mathcal{U}$ of $\boldsymbol{u^{\star}}$, which satisfies If such a function $V$ exists, then it is called a Lyapunov function, and $\boldsymbol{u^{\star}}$ is asymptotically stable. Moreover, $\boldsymbol{u^{\star}}$ is e

Figures (4)

  • Figure 1: The scratch of the architecture of the LyaDEQ model. The blue arrow represents a state that locally satisfies the Lyapunov exponential stability condition.
  • Figure 2: The architecture of our used ICNN.
  • Figure 3: t-SNE visualization results on the features after the orthogonal FC layer. 'adv' means test on the adversarial dataset.
  • Figure 4: Comparison between the DEQ model, the LyaDEQ model, and conventional CNNs on CIFAR10.

Theorems & Definitions (3)

  • Definition 1: Lyapunov stability
  • Theorem 2: Lyapunov stability theorem
  • proof