Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models

Kai Hu, Haoqi Hu, Matt Fredrikson

TL;DR

LipNeXt tackles the scalability barrier of Lipschitz-based certified robustness by introducing a constraint-free optimization of orthogonal matrices on the Stiefel manifold and a convolution-free Spatial Shift Module for effective spatial mixing. It scales to billion-parameter models and delivers state-of-the-art clean and certified robust accuracy (CRA) across CIFAR-10/100, Tiny-ImageNet, and ImageNet, including significant gains at radii such as $\varepsilon=1$. The method employs a norm-adaptive FastExp approximation with periodic polar retractions to maintain orthogonality efficiently while enabling low-precision training. Overall, LipNeXt demonstrates that deterministic, Lipschitz-based certification can scale with modern deep-learning trends without sacrificing guarantees or efficiency.

Abstract

Lipschitz-based certification offers efficient, deterministic robustness guarantees but has struggled to scale in model size, training efficiency, and ImageNet performance. We introduce \emph{LipNeXt}, the first \emph{constraint-free} and \emph{convolution-free} 1-Lipschitz architecture for certified robustness. LipNeXt is built using two techniques: (1) a manifold optimization procedure that updates parameters directly on the orthogonal manifold and (2) a \emph{Spatial Shift Module} to model spatial pattern without convolutions. The full network uses orthogonal projections, spatial shifts, a simple 1-Lipschitz $β$-Abs nonlinearity, and $L_2$ spatial pooling to maintain tight Lipschitz control while enabling expressive feature mixing. Across CIFAR-10/100 and Tiny-ImageNet, LipNeXt achieves state-of-the-art clean and certified robust accuracy (CRA), and on ImageNet it scales to 1-2B large models, improving CRA over prior Lipschitz models (e.g., up to $+8\%$ at $\varepsilon{=}1$) while retaining efficient, stable low-precision training. These results demonstrate that Lipschitz-based certification can benefit from modern scaling trends without sacrificing determinism or efficiency.

LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models

TL;DR

LipNeXt tackles the scalability barrier of Lipschitz-based certified robustness by introducing a constraint-free optimization of orthogonal matrices on the Stiefel manifold and a convolution-free Spatial Shift Module for effective spatial mixing. It scales to billion-parameter models and delivers state-of-the-art clean and certified robust accuracy (CRA) across CIFAR-10/100, Tiny-ImageNet, and ImageNet, including significant gains at radii such as . The method employs a norm-adaptive FastExp approximation with periodic polar retractions to maintain orthogonality efficiently while enabling low-precision training. Overall, LipNeXt demonstrates that deterministic, Lipschitz-based certification can scale with modern deep-learning trends without sacrificing guarantees or efficiency.

Abstract

Lipschitz-based certification offers efficient, deterministic robustness guarantees but has struggled to scale in model size, training efficiency, and ImageNet performance. We introduce \emph{LipNeXt}, the first \emph{constraint-free} and \emph{convolution-free} 1-Lipschitz architecture for certified robustness. LipNeXt is built using two techniques: (1) a manifold optimization procedure that updates parameters directly on the orthogonal manifold and (2) a \emph{Spatial Shift Module} to model spatial pattern without convolutions. The full network uses orthogonal projections, spatial shifts, a simple 1-Lipschitz -Abs nonlinearity, and spatial pooling to maintain tight Lipschitz control while enabling expressive feature mixing. Across CIFAR-10/100 and Tiny-ImageNet, LipNeXt achieves state-of-the-art clean and certified robust accuracy (CRA), and on ImageNet it scales to 1-2B large models, improving CRA over prior Lipschitz models (e.g., up to at ) while retaining efficient, stable low-precision training. These results demonstrate that Lipschitz-based certification can benefit from modern scaling trends without sacrificing determinism or efficiency.
Paper Structure (31 sections, 2 theorems, 39 equations, 8 tables, 2 algorithms)

This paper contains 31 sections, 2 theorems, 39 equations, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Preliminary and Related Work
  4. Preliminaries
  5. Lipschitz-based Method for Certified Robustness
  6. Orthogonal Manifold Optimization
  7. Related Work
  8. Re-parameterization for Lipschitz-based Certified Robustness
  9. Efficient Manifold Optimization for Orthogonality
  10. Method
  11. Constraint-Free Learning of Orthogonal Matrices on the Manifold
  12. Relation to skew re-parameterization
  13. Spatial Shift Module: A Convolution-Free Design
  14. 1D formulation.
  15. Theoretical justification.
  16. ...and 16 more sections

Key Result

Theorem 1

Let ${X}\in\mathbb{R}^{H\times W}$ be a single-channel tensor and $f_K$ be spatial convolution with kernel $K\in\mathbb{R}^{k\times k}$, unit stride, and circular padding (crucially relies on circular padding and does not hold under zero-padding). The operator $f_K$ is norm-preserving (tight 1-Lipsc if and only if kernel $K$ contains exactly one non-zero element with value $\pm 1$.

Theorems & Definitions (2)

  • Theorem 1
  • Theorem 2