Communication-Efficient Federated Learning by Exploiting Spatio-Temporal Correlations of Gradients

TL;DR

This paper tackles the communication bottleneck in Federated Learning by uncovering strong temporal correlations in client gradients across rounds and heterogeneous layer-wise parameter significance. It introduces GradESTC, a compression framework that represents gradients using a low-rank basis (captured by an orthonormal matrix $M$ and coefficients $A$) and maintains the basis over rounds with selective, incremental updates guided by temporal correlations. The method achieves substantial uplink reductions by transmitting lightweight coefficients and only a subset of updated basis vectors, while experiments show near-FedAvg convergence speed and accuracy across IID and non-IID settings. Theoretical convergence guarantees are provided under a probabilistic gradient-subspace assumption and inter-client error correlations, and empirical results demonstrate robust performance across multiple datasets and architectures, highlighting practical scalability for bandwidth-constrained FL deployments.

Abstract

Communication overhead is a critical challenge in federated learning, particularly in bandwidth-constrained networks. Although many methods have been proposed to reduce communication overhead, most focus solely on compressing individual gradients, overlooking the temporal correlations among them. Prior studies have shown that gradients exhibit spatial correlations, typically reflected in low-rank structures. Through empirical analysis, we further observe a strong temporal correlation between client gradients across adjacent rounds. Based on these observations, we propose GradESTC, a compression technique that exploits both spatial and temporal gradient correlations. GradESTC exploits spatial correlations to decompose each full gradient into a compact set of basis vectors and corresponding combination coefficients. By exploiting temporal correlations, only a small portion of the basis vectors need to be dynamically updated in each round. GradESTC significantly reduces communication overhead by transmitting lightweight combination coefficients and a limited number of updated basis vectors instead of the full gradients. Extensive experiments show that, upon reaching a target accuracy level near convergence, GradESTC reduces uplink communication by an average of 39.79% compared to the strongest baseline, while maintaining comparable convergence speed and final accuracy to uncompressed FedAvg. By effectively leveraging spatio-temporal gradient structures, GradESTC offers a practical and scalable solution for communication-efficient federated learning.

Figures (9)

• Figure 1: Cosine Similarity Heatmaps of the Client's Gradient Evolution. These heatmaps show the cosine similarity between a client's gradients in the first 40 rounds and specific rounds (5th, 10th, 15th, 20th, 25th, 30th). Darker shades (towards red) indicate higher similarity, and lighter shades (towards white) lower similarity. The deepest red columns indicate self-comparison (cosine similarity = 1). The x-axis represents global rounds, and the y-axis represents model layers, with shallow layers near the raw input.
• Figure 2: Parameter Size of Each Layer in ResNet18. The x-axis represents the layer index, while the y-axis indicates the parameter size in each layer.
• Figure 3: Gradient flattening with WHDC ordering.
• Figure 4: Experimental Results of Different Algorithms. Solid bars indicate total uplink overhead, diagonal bars show overhead to reach $x$% accuracy, red dots mark the peak accuracy achieved, and the red horizontal dotted line signifies the specified convergence accuracy threshold.
• Figure 5: Test Accuracy vs. Overhead. GradESTC achieves the highest accuracy with minimal communication overhead.

Theorems & Definitions (7)

• Theorem 1
• Theorem 2
• Corollary 1
• Proof 1
• Lemma 1
• Proof 2
• Proof 3