PID-Guided Partial Alignment for Multimodal Decentralized Federated Learning

TL;DR

The paper tackles multimodal decentralized federated learning under modality heterogeneity and absence of a central server. It introduces PARSE, a PID-guided framework that factorizes each modality's latent representation into redundant $z^{r}$, unique $z^{u}$, and synergistic $z^{s}$ slices, enabling selective slice-level sharing via per-modality subgraphs. By aligning only the shareable slices and keeping non-shared components local, PARSE mitigates uni-/multimodal gradient conflicts and explicitly leverages cross-modal synergy for multimodal agents. Across four benchmarks and diverse agent mixes, PARSE yields consistent improvements over baselines, with ablations confirming robust performance across feature-split ratios, fusion methods, and non-IID conditions.

Abstract

Multimodal decentralized federated learning (DFL) is challenging because agents differ in available modalities and model architectures, yet must collaborate over peer-to-peer (P2P) networks without a central coordinator. Standard multimodal pipelines learn a single shared embedding across all modalities. In DFL, such a monolithic representation induces gradient misalignment between uni- and multimodal agents; as a result, it suppresses heterogeneous sharing and cross-modal interaction. We present PARSE, a multimodal DFL framework that operationalizes partial information decomposition (PID) in a server-free setting. Each agent performs feature fission to factorize its latent representation into redundant, unique, and synergistic slices. P2P knowledge sharing among heterogeneous agents is enabled by slice-level partial alignment: only semantically shareable branches are exchanged among agents that possess the corresponding modality. By removing the need for central coordination and gradient surgery, PARSE resolves uni-/multimodal gradient conflicts, thereby overcoming the multimodal DFL dilemma while remaining compatible with standard DFL constraints. Across benchmarks and agent mixes, PARSE yields consistent gains over task-, modality-, and hybrid-sharing DFL baselines. Ablations on fusion operators and split ratios, together with qualitative visualizations, further demonstrate the efficiency and robustness of the proposed design.

Figures (8)

• Figure 1: We study three knowledge-sharing strategies in DFL setting. To illustrate, we consider a two-modality scenario involving three types of agents: Red: modality A only; yellow: modality B only; bicolored: both.
• Figure 2: Test accuracy for unimodal and multimodal agents under task-, modality-, and hybrid sharing across three agent mixes: 6 audio, 6 video, and 18 multimodal agents; (b) 10 audio, 10 video, and 10 multimodal agents; and (c) 13 audio, 13 video, and 4 multimodal agents. [A], [V], [AV] are averages over audio-only, video-only, and multimodal agents.
• Figure 3: Group agents by their modality subsets, we show inter-group gradient cosine similarity (w.r.t. shared models) and intra-group gradient cosine similarity.
• Figure 4: PARSE at a glance (two modalities for illustration). Agents that share a modality form a modality-specific P2P subgraph. Each encoder output is fissioned into redundant ($z^{r}$), synergistic ($z^{s}$), and unique ($z^{u}$) slices. Unimodal agents train on $z^{u}$ and $z^{r}$. Multimodal agents: (i) partially align$z^{u}$ and $z^{r}$ across agents, and (ii) learn a multimodal classifier on the fused $z^{s}$. This routes shareable information while keeping modality-exclusive and joint-only components local.
• Figure 5: t-SNE visualization of synergistic features from each modality and after fusion.