Federated Learning Using Coupled Tensor Train Decomposition

TL;DR

Inspired by the efficient tensor train decomposition, this work proposes a coupled tensor train (CTT) decomposition for federated learning that can extract common features of coupled modes while maintaining the different features of uncoupled modes.

Abstract

Coupled tensor decomposition (CTD) can extract joint features from multimodal data in various applications. It can be employed for federated learning networks with data confidentiality. Federated CTD achieves data privacy protection by sharing common features and keeping individual features. However, traditional CTD schemes based on canonical polyadic decomposition (CPD) may suffer from low computational efficiency and heavy communication costs. Inspired by the efficient tensor train decomposition, we propose a coupled tensor train (CTT) decomposition for federated learning. The distributed coupled multi-way data are decomposed into a series of tensor trains with shared factors. In this way, we can extract common features of coupled modes while maintaining the different features of uncoupled modes. Thus the privacy preservation of information across different network nodes can be ensured. The proposed CTT approach is instantiated for two fundamental network structures, namely master-slave and decentralized networks. Experimental results on synthetic and real datasets demonstrate the superiority of the proposed schemes over existing methods in terms of both computational efficiency and communication rounds. In a classification task, experimental results show that the CTT-based federated learning achieves almost the same accuracy performance as that of the centralized counterpart.

Figures (15)

• Figure 1: 3rd-order tensor coupling across the first mode (left) and across the first and second modes (right).
• Figure 2: Tensor network illustration of the contraction of two tensors of size $I_1\times\cdots\times I_N\times J_1\times\cdots\times J_L$ and $J_1\times\cdots\times J_L \times K_1\times\cdots\times K_M$.
• Figure 3: Tensor network illustration of the TT decomposition for an $I_1\times I_2\times\cdots\times I_N$ tensor.
• Figure 4: The network architecture of CTT. $\mathcal{X}^k$: the tensor stored in the $k$th client. $\mathcal{G}_1^k$: the TT-core for the personal mode of the $k$th client. $\mathcal{G}_n$: the TT-core for the $n$th global feature mode, $n=2,\ldots,N$.
• Figure 5: Illustration of a master-slave network structure.

Theorems & Definitions (4)

• Definition 1: Tensor sdfhpf17
• Definition 2: $n$-unfolding sdfhpf17
• Definition 3: Tensor contraction product sdfhpf17
• Definition 4: Tensor Train Decomposition (TTD) oseledets2011tensor