THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compression

TL;DR

THC tackles the bottleneck of gradient communication in distributed deep learning by introducing Tensor Homomorphic Compression, a bi-directional compression framework that allows direct aggregation of compressed gradients and supports in-network aggregation. By enforcing Uniform (and Non-uniform) Homomorphic Compression, augmented with Randomized Hadamard Transform preprocessing, THC achieves high-accuracy gradient averaging with significantly reduced bandwidth and computation at the parameter server, enabling switch-based acceleration. The authors present a comprehensive implementation on BytePS with both software and programmable-switch PS, plus an offline-optimized lookup-table design, and demonstrate substantial end-to-end speedups (up to 1.47x TTA) and throughput gains on vision and language models across local hardware and AWS EC2. These results indicate THC’s practical potential for scaling distributed DL in data-center networks and its compatibility with evolving network acceleration hardware.

Abstract

Deep neural networks (DNNs) are the de facto standard for essential use cases, such as image classification, computer vision, and natural language processing. As DNNs and datasets get larger, they require distributed training on increasingly larger clusters. A main bottleneck is the resulting communication overhead where workers exchange model updates (i.e., gradients) on a per-round basis. To address this bottleneck and accelerate training, a widely-deployed approach is compression. However, previous deployments often apply bi-directional compression schemes by simply using a uni-directional gradient compression scheme in each direction. This results in significant computational overheads at the parameter server and increased compression error, leading to longer training and lower accuracy. We introduce Tensor Homomorphic Compression (THC), a novel bi-directional compression framework that enables the direct aggregation of compressed values and thus eliminating the aforementioned computational overheads. Moreover, THC is compatible with in-network aggregation (INA), which allows for further acceleration. Our evaluation shows that training representative vision and language models with THC reaches target accuracy by 1.40x to 1.47x faster using INA and 1.28x to 1.33x faster using a software PS compared with state-of-the-art systems.

Figures (16)

• Figure 1: End-to-end workflow when using typical compression algorithms.
• Figure 2: Communication time and error.
• Figure 3: An illustration of THC, with the section numbers of each component.
• Figure 4: THC's workflow illustration.
• Figure 5: Time to accuracy (TTA) comparison over one image processing task (i.e., VGG16) and two NLP tasks (i.e., GPT-2 and RoBERTa-base). The second row zooms in on the competitive results.

Theorems & Definitions (6)

• Definition 1: Uniform Homomorphic Compression
• Definition 2: Homomorphic USQ
• Definition 3: Non-uniform homomorphic compression
• Definition 4: compression operator
• Definition 5: decompression operator
• Definition 6: Uniform Homomorphic Compression