Stochastic Difference-of-Convex Optimization with Momentum

TL;DR

This paper addresses stochastic DC optimization with $f({\bm{x}}) = g({\bm{x}}) - h({\bm{x}})$ under small-batch, noisy gradients. It argues that momentum is necessary for convergence in realistic settings and introduces both double-loop and single-loop momentum-based SPDC variants, along with a momentum-variance reduction enhancement. Theoretical results show convergence guarantees and rates under standard smoothness and bounded-variance assumptions, along with a lower-bound construction illustrating failures of momentum-free methods. Empirical results corroborate the robustness of the momentum-based approaches in noisy, small-batch regimes. The work has practical implications for training non-convex models with DC decompositions without resorting to large mini-batches.

Abstract

Stochastic difference-of-convex (DC) optimization is prevalent in numerous machine learning applications, yet its convergence properties under small batch sizes remain poorly understood. Existing methods typically require large batches or strong noise assumptions, which limit their practical use. In this work, we show that momentum enables convergence under standard smoothness and bounded variance assumptions (of the concave part) for any batch size. We prove that without momentum, convergence may fail regardless of stepsize, highlighting its necessity. Our momentum-based algorithm achieves provable convergence and demonstrates strong empirical performance.

Figures (2)

• Figure 1: Effect of increasing stochastic noise ($\sigma$) on the performance of double-loop SPDC with and without momentum. We fix $a = 0.9$ and sweep over $\gamma$ and $\alpha$ (for momentum). Momentum ensures stability and convergence as noise increases, whereas the non-momentum variant quickly degrades.
• Figure 2: Effect of increasing stochastic noise ($\sigma$) on the performance of single-loop SPDC with and without momentum. We fix $\gamma = 0.01$ and $a = 0.9$, and sweep over $\eta_0$ while keeping $\eta_1 = 0.01$. Momentum significantly improves robustness across all noise levels.

Theorems & Definitions (8)

• proof
• proof
• proof
• proof
• proof
• proof
• proof
• proof