Flattened one-bit stochastic gradient descent: compressed distributed optimization with controlled variance
Alexander Stollenwerk, Laurent Jacques
TL;DR
The paper introduces FO-SGD, a distributed SGD algorithm that compresses gradients in both worker-to-server and server-to-worker directions by combining a dithering-based one-bit quantizer with a randomized universal sensing basis to flatten gradients before quantization. This flattening mitigates variance blow-up and enables robust convergence guarantees even for sparse stochastic gradients, while remaining computationally efficient via the fast Walsh-Hadamard transform. The authors provide convergence guarantees for convex objectives and establish non-convex convergence rates to stationary points, with error terms controlled by quantization and transform parameters. The approach achieves full bidirectional compression and is well-suited for scalable distributed optimization in large-scale learning settings.
Abstract
We propose a novel algorithm for distributed stochastic gradient descent (SGD) with compressed gradient communication in the parameter-server framework. Our gradient compression technique, named flattened one-bit stochastic gradient descent (FO-SGD), relies on two simple algorithmic ideas: (i) a one-bit quantization procedure leveraging the technique of dithering, and (ii) a randomized fast Walsh-Hadamard transform to flatten the stochastic gradient before quantization. As a result, the approximation of the true gradient in this scheme is biased, but it prevents commonly encountered algorithmic problems, such as exploding variance in the one-bit compression regime, deterioration of performance in the case of sparse gradients, and restrictive assumptions on the distribution of the stochastic gradients. In fact, we show SGD-like convergence guarantees under mild conditions. The compression technique can be used in both directions of worker-server communication, therefore admitting distributed optimization with full communication compression.