Beyond Gradient Averaging in Parallel Optimization: Improved Robustness through Gradient Agreement Filtering

TL;DR

The paper tackles the problem that gradient averaging in distributed SGD can be destabilizing when micro-gradients disagree, especially late in training or under label noise. It introduces Gradient Agreement Filtering (GAF), which filters micro-gradients based on cosine-distance similarity before averaging to form the macrobatch gradient, thereby improving robustness and generalization. Empirical results on CIFAR-100 and CIFAR-100N-Fine show consistent validation-accuracy gains, with larger improvements under higher label-noise levels and at smaller microbatch sizes, at times reducing compute by enabling smaller microbatches without sacrificing stability. The approach yields up to 18.4% absolute gains under heavy noise and demonstrates that GAF can maintain or improve performance with substantially reduced compute, highlighting its practical potential for robust distributed training. The work also outlines concrete future directions, including adaptive thresholds, alternative similarity metrics, and extensions to other domains and optimizers.

Abstract

We introduce Gradient Agreement Filtering (GAF) to improve on gradient averaging in distributed deep learning optimization. Traditional distributed data-parallel stochastic gradient descent involves averaging gradients of microbatches to calculate a macrobatch gradient that is then used to update model parameters. We find that gradients across microbatches are often orthogonal or negatively correlated, especially in late stages of training, which leads to memorization of the training set, reducing generalization. In this paper, we introduce a simple, computationally effective way to reduce gradient variance by computing the cosine distance between micro-gradients during training and filtering out conflicting updates prior to averaging. We improve validation accuracy with significantly smaller microbatch sizes. We also show this reduces memorizing noisy labels. We demonstrate the effectiveness of this technique on standard image classification benchmarks including CIFAR-100 and CIFAR-100N-Fine. We show this technique consistently outperforms validation accuracy, in some cases by up to 18.2\% compared to traditional training approaches while reducing the computation required nearly an order of magnitude because we can now rely on smaller microbatch sizes without destabilizing training.

Figures (9)

• Figure 1: We train a ResNet18 on CIFAR-100 (left) and CIFAR-100N-Fine (right). We show train and validation accuracy over iterations (top). The rolling average of cosine distances is shown in dark green and the raw cosine distances during training in light green (bottom). In late stages of training in all runs, as training accuracy plateaus, the cosine distance between micro-gradients approaches 1 with many micro-gradients diverging even further up to 1.1 for CIFAR-100 and 1.6 for CIFAR-100N-Fine.
• Figure 2: Visualization of cosine distance between micro-gradient batches in 2D. Aligned gradients have low cosine distance (left), while orthogonal or negatively correlated gradients have high cosine distance (right).
• Figure 3: Cosine distances between micro-gradients during the later stages of a baseline ViT-L/16 run on ILSVRC12
• Figure 4: Rolling average of cosine distances between micro-gradients during 10 baseline runs on CIFAR-100 varying batch sizes from 100 to 1000. As the microbatch size increases, the micro-gradients become more and more correlated throughout training.
• Figure 5: Train and validation accuracy (top) and the cosine distance between micro-gradients (bottom) with rolling average in dark green and raw values in light green, over iterations of a baseline training ResNet18 without GAF on random noise. The model overfits, reaching 100% training accuracy, but the micro-gradients cosine distance remains above 0.96 throughout the entire training, and above 0.99 for all iterations after the very early iterations.