Dynamic Network-Assisted D2D-Aided Coded Distributed Learning

TL;DR

This work designs a novel D2D-aided coded distributed learning method named D 2D-CDL for efficient load balancing across devices and derives an optimal compression rate, which minimizes the processing time.

Abstract

Today, various machine learning (ML) applications offer continuous data processing and real-time data analytics at the edge of a wireless network. Distributed real-time ML solutions are highly sensitive to the so-called straggler effect caused by resource heterogeneity and alleviated by various computation offloading mechanisms that seriously challenge the communication efficiency, especially in large-scale scenarios. To decrease the communication overhead, we rely on device-to-device (D2D) connectivity that improves spectrum utilization and allows efficient data exchange between devices in proximity. In particular, we design a novel D2D-aided coded federated learning method (D2D-CFL) for efficient load balancing across devices. The proposed solution captures system dynamics, including data (time-dependent learning model, varied intensity of data arrivals), device (diverse computational resources and volume of training data), and deployment (varied locations and D2D graph connectivity). To minimize the number of communication rounds, we derive an optimal compression rate for achieving minimum processing time and establish its connection with the convergence time. The resulting optimization problem provides suboptimal compression parameters, which improve the total training time. Our proposed method is beneficial for real-time collaborative applications, where the users continuously generate training data resulting in the model drift.

Figures (10)

• Figure 1: Illustration of D2D-aided distributed learning scenario, where two devices train collaborative model iteratively by exchanging updates with central server that aggregates partial gradients of all data points and orchestrates distributed learning. Device with lower computational capacity may offload excessive data to its stronger neighbor to accelerate training process.
• Figure 2: Time diagram for one iteration of D2D-CFL.
• Figure 3: Comparison of baseline FL and D2D-CFL in time and iterations. Static model $\boldsymbol{\beta}^*$: normalized error $\frac{||\boldsymbol{\beta}^{(t)} - \boldsymbol{\beta}^*||_2}{|| \boldsymbol{\beta}^*||_2}$ for equal compression rate $\gamma_i$. Improvement is up to $8\%$.
• Figure 4: Comparison of baseline FL and D2D-CFL in time and iterations. Static model $\boldsymbol{\beta}^*$: normalized error $\frac{||\boldsymbol{\beta}^{(t)} - \boldsymbol{\beta}^*||_2}{|| \boldsymbol{\beta}^*||_2}$ for optimal compression rate $\gamma_i$. Improvement is up to $50\%$.
• Figure 5: Comparison of baseline FL and D2D-CFL in time and iterations. Dynamic model $\boldsymbol{\beta}^*$: normalized error $\frac{||\boldsymbol{\beta}^{(t)} - \boldsymbol{\beta}^*||_2}{|| \boldsymbol{\beta}^*||_2}$ for optimal compression rate $\gamma_i$. Improvement is up to $50\%$.

Theorems & Definitions (20)

• Definition 1: $\varepsilon$-MI-DP, cuff2016differential
• Remark 1
• Lemma 1: D2D-CFL optimal compression rate
• proof
• Lemma 2: D2D-CFL direct link privacy budget
• proof
• Corollary 1
• proof
• Remark 2
• Lemma 3: D2D-CFL achievable deadline