FSSUAVL: A Discriminative Framework using Vision Models for Federated Self-Supervised Audio and Image Understanding
Yasar Abbas Ur Rehman, Kin Wai Lau, Yuyang Xie, Ma Lan, JiaJun Shen
TL;DR
FSSUAVL introduces a single-model, federated self-supervised learning framework that discriminates unpaired audio and image data by projecting them into a shared embedding space, eliminating the need for modality alignment. It leverages sequential contrastive SSL (NT-Xent) on a common encoder, enabling both unimodal and multimodal downstream tasks while maintaining data privacy in cross-device FL via FedAvg. The method demonstrates superior or competitive performance across vision and audio benchmarks, including robust cross-domain transfer to out-of-domain datasets, and shows efficiency advantages over combining separate modality-specific models. The approach is validated with CNN and ViT encoders, using unpaired data distributed across non-IID clients, and opens avenues for extending to additional modalities in future work.
Abstract
Recent studies have demonstrated that vision models can effectively learn multimodal audio-image representations when paired. However, the challenge of enabling deep models to learn representations from unpaired modalities remains unresolved. This issue is especially pertinent in scenarios like Federated Learning (FL), where data is often decentralized, heterogeneous, and lacks a reliable guarantee of paired data. Previous attempts tackled this issue through the use of auxiliary pretrained encoders or generative models on local clients, which invariably raise computational cost with increasing number modalities. Unlike these approaches, in this paper, we aim to address the task of unpaired audio and image recognition using \texttt{FSSUAVL}, a single deep model pretrained in FL with self-supervised contrastive learning (SSL). Instead of aligning the audio and image modalities, \texttt{FSSUAVL} jointly discriminates them by projecting them into a common embedding space using contrastive SSL. This extends the utility of \texttt{FSSUAVL} to paired and unpaired audio and image recognition tasks. Our experiments with CNN and ViT demonstrate that \texttt{FSSUAVL} significantly improves performance across various image- and audio-based downstream tasks compared to using separate deep models for each modality. Additionally, \texttt{FSSUAVL}'s capacity to learn multimodal feature representations allows for integrating auxiliary information, if available, to enhance recognition accuracy.