Deterministic Nonsmooth Nonconvex Optimization

TL;DR

The paper investigates the complexity of obtaining $$(\delta,\epsilon)$$-Goldstein stationary points for Lipschitz, potentially nonsmooth, nonconvex objectives using deterministic and randomized first-order methods. It shows randomness is essential for dimension-free rates, establishing a linear-in-$d$ lower bound for deterministic methods and demonstrating that zeroth-order access is necessary for finite-time guarantees, while also presenting a deterministic approach with a mild logarithmic dependence on smoothness for slightly smooth functions. It further proves that no deterministic, black-box smoothing can yield dimension-free improvements, but introduces a practical white-box smoothing technique for ReLU networks (via neural arithmetic circuits) that preserves stationary points and achieves a dimension-free algorithm in many common architectures. Collectively, the results delineate the boundaries between randomness, smoothing, and architecture-informed strategies in nonsmooth nonconvex optimization and provide a first deterministic dimension-free path for optimizing several ReLU-based models under structured access.

Abstract

We study the complexity of optimizing nonsmooth nonconvex Lipschitz functions by producing $(δ,ε)$-stationary points. Several recent works have presented randomized algorithms that produce such points using $\tilde O(δ^{-1}ε^{-3})$ first-order oracle calls, independent of the dimension $d$. It has been an open problem as to whether a similar result can be obtained via a deterministic algorithm. We resolve this open problem, showing that randomization is necessary to obtain a dimension-free rate. In particular, we prove a lower bound of $Ω(d)$ for any deterministic algorithm. Moreover, we show that unlike smooth or convex optimization, access to function values is required for any deterministic algorithm to halt within any finite time. On the other hand, we prove that if the function is even slightly smooth, then the dimension-free rate of $\tilde O(δ^{-1}ε^{-3})$ can be obtained by a deterministic algorithm with merely a logarithmic dependence on the smoothness parameter. Motivated by these findings, we turn to study the complexity of deterministically smoothing Lipschitz functions. Though there are efficient black-box randomized smoothings, we start by showing that no such deterministic procedure can smooth functions in a meaningful manner, resolving an open question. We then bypass this impossibility result for the structured case of ReLU neural networks. To that end, in a practical white-box setting in which the optimizer is granted access to the network's architecture, we propose a simple, dimension-free, deterministic smoothing that provably preserves $(δ,ε)$-stationary points. Our method applies to a variety of architectures of arbitrary depth, including ResNets and ConvNets. Combined with our algorithm, this yields the first deterministic dimension-free algorithm for optimizing ReLU networks, circumventing our lower bound.

Theorems & Definitions (20)

• Theorem 3.1
• Theorem 3.2
• Remark 3.1
• Remark 3.2
• Theorem 4.1
• Definition 5.1
• Theorem 5.1
• proof
• Definition 5.2: Neural Arithmetic Circuits (NAC)
• Example 5.1