Optimal lower Lipschitz bounds for ReLU layers, saturation, and phase retrieval
Daniel Freeman, Daniel Haider
TL;DR
The paper addresses stable, injective recovery from non-linear measurements formed by a linear frame followed by a component-wise activation, encompassing ReLU layers, $\lambda$-saturation, and phase retrieval. It develops a unified frame-theoretic framework in which injectivity reduces to activated subframes and derives dimension-free lower Lipschitz bounds $\kappa_L$ expressed via $A_\alpha$, $A_\lambda$, and related quantities; for ReLU and saturation, explicit bounds are $\tfrac{1}{2}\sqrt{A_\alpha} \le \kappa_L \le \sqrt{A_\alpha}$ and $\min\{\tfrac{1}{2}\sqrt{A_\lambda}, \lambda\} \le \kappa_L \le \sqrt{A_\lambda}$, respectively. The work also revisits phase retrieval with improved stability bounds based on the $\sigma$-strong complement property, yielding $\sqrt{A_{|\cdot|}} \le \kappa_L \le \sqrt{2}\sigma$ and $\sigma^2 \le A_{|\cdot|} \le 2\sigma^2$. By unifying these three problems under a common non-linear measurement framework, the results provide rigorous, dimension-free stability guarantees for robust recovery in saturated devices, ReLU networks, and phase retrieval, and address open questions about critical thresholds and constants in these settings.
Abstract
The injectivity of ReLU layers in neural networks, the recovery of vectors from clipped or saturated measurements, and (real) phase retrieval in $\mathbb{R}^n$ allow for a similar problem formulation and characterization using frame theory. In this paper, we revisit all three problems with a unified perspective and derive lower Lipschitz bounds for ReLU layers and clipping which are analogous to the previously known result for phase retrieval and are optimal up to a constant factor.