Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Why Smooth Stability Assumptions Fail for ReLU Learning

Ronald Katende

TL;DR

The paper shows that global smoothness-based stability proxies fail for ReLU learning because activation boundary crossings induce nonlocal changes in the gradient. It provides a concrete one-neuron counterexample to demonstrate the nonexistence of a uniform $L$- Lipschitz bound on the gradient and argues that a nonsmooth framework based on Clarke generalized Jacobians is the correct lens for stability analysis. By linking the obstruction to boundary geometry and showing that surrogate smooth activations require diverging constants, it clarifies why smooth approximations can be misleading. The work offers a precise, minimal nonsmooth regularity condition under which stability statements can be meaningfully formulated, guiding the development of nonsmooth-aware stability theories for ReLU-based networks.

Abstract

Stability analyses of modern learning systems are frequently derived under smoothness assumptions that are violated by ReLU-type nonlinearities. In this note, we isolate a minimal obstruction by showing that no uniform smoothness-based stability proxy such as gradient Lipschitzness or Hessian control can hold globally for ReLU networks, even in simple settings where training trajectories appear empirically stable. We give a concrete counterexample demonstrating the failure of classical stability bounds and identify a minimal generalized derivative condition under which stability statements can be meaningfully restored. The result clarifies why smooth approximations of ReLU can be misleading and motivates nonsmooth-aware stability frameworks.

Why Smooth Stability Assumptions Fail for ReLU Learning

TL;DR

The paper shows that global smoothness-based stability proxies fail for ReLU learning because activation boundary crossings induce nonlocal changes in the gradient. It provides a concrete one-neuron counterexample to demonstrate the nonexistence of a uniform - Lipschitz bound on the gradient and argues that a nonsmooth framework based on Clarke generalized Jacobians is the correct lens for stability analysis. By linking the obstruction to boundary geometry and showing that surrogate smooth activations require diverging constants, it clarifies why smooth approximations can be misleading. The work offers a precise, minimal nonsmooth regularity condition under which stability statements can be meaningfully formulated, guiding the development of nonsmooth-aware stability theories for ReLU-based networks.

Abstract

Stability analyses of modern learning systems are frequently derived under smoothness assumptions that are violated by ReLU-type nonlinearities. In this note, we isolate a minimal obstruction by showing that no uniform smoothness-based stability proxy such as gradient Lipschitzness or Hessian control can hold globally for ReLU networks, even in simple settings where training trajectories appear empirically stable. We give a concrete counterexample demonstrating the failure of classical stability bounds and identify a minimal generalized derivative condition under which stability statements can be meaningfully restored. The result clarifies why smooth approximations of ReLU can be misleading and motivates nonsmooth-aware stability frameworks.
Paper Structure (11 sections, 3 theorems, 22 equations, 1 figure)

This paper contains 11 sections, 3 theorems, 22 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Setup and Notation
  3. Smoothness obstructions and a nonsmooth substitute
  4. Setting and smoothness proxy under test
  5. Nonexistence of a global Lipschitz constant for the gradient map
  6. Takeaway (global vs. local).
  7. Why smooth surrogates do not transfer uniformly to ReLU
  8. A minimal nonsmooth regularity that matches the geometry
  9. Diagnostic message.
  10. Implications
  11. Conclusion

Key Result

Theorem 1

Fix any $x\neq 0$ and any $y\neq 0$. For $\mathcal{L}$ defined by eq:model-1d--eq:loss, there is no finite $L$ such that eq:L-smooth holds on $\mathbb{R}^2$. In particular, $\theta\mapsto\nabla \mathcal{L}(\theta)$ is not globally Lipschitz, hence any argument that requires a uniform bound on $\|\na

Figures (1)

  • Figure 1: Schematic illustration of the smoothness obstruction for ReLU learning. An $O(\varepsilon)$ parameter perturbation across the activation boundary $wx+b=0$ induces an $O(1)$ jump in the gradient, ruling out any global Lipschitz or bounded-Hessian proxy.

Theorems & Definitions (7)

  • Theorem 1: Failure of global $L$-smoothness for ReLU parameterizations
  • proof
  • Remark 1: Structural nature of the obstruction
  • Proposition 1: Surrogate smoothness constants must diverge
  • proof
  • Proposition 2: A nonsmooth contraction criterion for perturbation stability
  • proof