A stochastic smoothing framework for nonconvex-nonconcave min-sum-max problems with applications to Wasserstein distributionally robust optimization
Wei Liu, Muhammad Khan, Gabriel Mancino-Ball, Yangyang Xu
TL;DR
This work tackles nonconvex-nonconcave min-sum-max problems that arise in adversarial training and WDRO. It introduces SSPG, a stochastic smoothing proximal gradient method that replaces the inner maximization with a log-sum-exp smoothing, yielding a differentiable surrogate $\widetilde{g}$ whose gradients can be estimated stochastically. The authors establish Clarke regularity of the primal objective, show equivalence to directional stationarity, and prove almost-sure convergence to Clarke stationary points with an $\tilde{O}(\epsilon^{-3})$ iteration complexity for obtaining an $\epsilon$-scaled stationary solution. Numerical experiments on Newsvendor, regression, and adversarial DL tasks demonstrate improved accuracy and robustness, with favorable time complexity relative to competing WDRO methods. Overall, the framework provides a theoretically grounded, scalable approach to WDRO and related nonconvex-nonconcave problems, enabling robust decision-making in high-dimensional learning settings.
Abstract
Applications such as adversarially robust training and Wasserstein Distributionally Robust Optimization (WDRO) can be naturally formulated as min-sum-max optimization problems. While this formulation can be rewritten as an equivalent min-max problem, the summation of max terms introduces computational challenges, including increased complexity and memory demands, which must be addressed. These challenges are particularly evident in WDRO, where existing tractable algorithms often rely on restrictive assumptions on the objective function, limiting their applicability to state-of-the-art machine learning problems such as the training of deep neural networks. This study introduces a novel stochastic smoothing framework based on the \mbox{log-sum-exp} function, efficiently approximating the max operator in min-sum-max problems. By leveraging the Clarke regularity of the max operator, we develop an iterative smoothing algorithm that addresses these computational difficulties and guarantees almost surely convergence to a Clarke/directional stationary point. We further prove that the proposed algorithm finds an $ε$-scaled Clarke stationary point of the original problem, with a worst-case iteration complexity of $\widetilde{O}(ε^{-3})$. Our numerical experiments demonstrate that our approach outperforms or is competitive with state-of-the-art methods in solving the newsvendor problem, deep learning regression, and adversarially robust deep learning. The results highlight that our method yields more accurate and robust solutions in these challenging problem settings.