Curvature accelerated decentralized non-convex optimization for high-dimensional machine learning problems
Dingran Yi, Fanhao Zeng, Nikolaos M. Freris
TL;DR
This work tackles decentralized non-convex optimization for high-dimensional machine learning by proposing CADEN, a curvature-accelerated primal-dual method. CADEN integrates a quasi-Newton (L-BFGS) local solver with a communication-efficient ADMM-style update, reducing the per-round communication to a single broadcast and supporting partial participation. The authors prove sublinear convergence to stationary points with a rate of $O(1/T)$ and analyze how the rate depends on local conditioning, network topology, and agent activity. Empirical results on distributed MNIST neural network training show faster convergence and substantial reductions in both computation and communication relative to strong baselines, confirming practical impact in heterogeneous environments.
Abstract
We consider distributed optimization as motivated by machine learning in a multi-agent system: each agent holds local data and the goal is to minimize an aggregate loss function over a common model, via an interplay of local training and distributed communication. In the most interesting case of training a neural network, the loss functions are non-convex and the high dimension of the model poses challenges in terms of communication and computation. We propose a primal-dual method that leverages second order information in the local training sub-problems in order to accelerate the algorithm. To ease the computational burden, we invoke a quasi-Newton local solver with linear cost in the model dimension. Besides, our method is communication efficient in the sense of requiring to broadcast the local model only once per round. We rigorously establish the convergence of the algorithm and demonstrate its merits by numerical experiments.