Softmax is $1/2$-Lipschitz: A tight bound across all $\ell_p$ norms

TL;DR

This work establishes a universal $\tfrac{1}{2}$-Lipschitz bound for the softmax operator across all $\ell_p$ norms ($p \ge 1$), derived via the softmax Jacobian ${\mathbf J}_{\sigma_{\lambda}}({\bm{x}})$ and a norm interpolation inequality. The bound is tight: the supremum of the Jacobian norm is $\tfrac{1}{2}$ for $p=1$ and $p=\infty$, and is approached for $p\in(1,\infty)$, yielding a norm-uniform result that sharpens previous assumptions of a unit Lipschitz constant. The authors demonstrate practical impact by refining existing robustness and convergence analyses, including improved constants for attention mechanisms and convergence guarantees in entropy-regularized double-softmax fixed-point problems. Empirically, the $1/2$ bound holds across large-scale vision, language, and reinforcement learning models, with several settings approaching the bound, confirming the bound’s tightness and broad applicability.

Abstract

The softmax function is a basic operator in machine learning and optimization, used in classification, attention mechanisms, reinforcement learning, game theory, and problems involving log-sum-exp terms. Existing robustness guarantees of learning models and convergence analysis of optimization algorithms typically consider the softmax operator to have a Lipschitz constant of $1$ with respect to the $\ell_2$ norm. In this work, we prove that the softmax function is contractive with the Lipschitz constant $1/2$, uniformly across all $\ell_p$ norms with $p \ge 1$. We also show that the local Lipschitz constant of softmax attains $1/2$ for $p = 1$ and $p = \infty$, and for $p \in (1,\infty)$, the constant remains strictly below $1/2$ and the supremum $1/2$ is achieved only in the limit. To our knowledge, this is the first comprehensive norm-uniform analysis of softmax Lipschitz continuity. We demonstrate how the sharper constant directly improves a range of existing theoretical results on robustness and convergence. We further validate the sharpness of the $1/2$ Lipschitz constant of the softmax operator through empirical studies on attention-based architectures (ViT, GPT-2, Qwen3-8B) and on stochastic policies in reinforcement learning.

Figures (4)

• Figure 1: Empirical $L_p$ of the softmax operator over attention scores from three Vision Transformer (ViT) variants on CIFAR-100 (top row) and ImageNet (bottom row), across varying perturbation magnitudes $\epsilon$ for multiple $\ell_p$ norms. In all cases, the empirical values remain below the derived bound of $1/2$.
• Figure 2: Empirical $L_p$ of the softmax operator for classification logits of ResNET50 model on (a) CIFAR-100 and (b) ImageNet dataset, across varying perturbation magnitudes $\epsilon$ for multiple $\ell_p$ norms. In all cases, the empirical values remain well below the derived bound of $1/2$.
• Figure 3: Empirical $L_p$ of the softmax operator over attention scores from two Large Language models, GPT-2 and QWEN-3, on Hellswag dataset (top row) and PIQA dataset (bottom row), across varying perturbation magnitudes $\epsilon$ for multiple $\ell_p$ norms. Across all configurations, the empirical values remain below the theoretical bound of $1/2$, with several instances approaching this limit, thereby confirming the tightness of the derived bound.
• Figure 4: Empirical Lipschitz sensitivity of the softmax policy in RL environments, Cartpole (top row) and Lunarlander (bottom row), across varying perturbation magnitudes $\epsilon$, under varying coefficients $\lambda$ for different $p$-norms. The empirical values scale with $\lambda$ and remain below the theoretical bound of $\lambda/2$, thereby empirically confirming the derived bound.

Theorems & Definitions (17)

• Definition 2.1: $\ell_p$ norm
• Definition 2.2: Lipschitz continuity in $\ell_p$ norm
• Definition 2.3: Jacobian matrix boyd2004convex
• Definition 2.4: Local Lipschitz constant rockafellar1998variational
• Lemma 1: Lipschitz constant via the Jacobian
• Lemma 2: Jacobian of the softmax
• Proposition 1: Norm Interpolation
• Lemma 3: Lipschitz constant of the softmax operator
• Theorem 1: Lipschitz constant of softmax function
• Proposition 2: Tightness of the Lipschitz constant for softmax