Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Federated Fine-Tuning of Self-Supervised Speech Representations

Xin Guo, Chunrui Zhao, Hong Jia, Ting Dang, Gongping Huang, Xianrui Zheng, Yan Gao

Abstract

Integrating Federated Learning (FL) with self-supervised learning (SSL) enables privacy-preserving fine-tuning for speech tasks. However, federated environments exhibit significant heterogeneity: clients differ in computational capacity, causing straggler effects under unified fine-tuning, while diverse downstream tasks require different representation depths, making full-model updates inefficient. To address these challenges, we propose an adaptive federated fine-tuning framework with early exits. Lightweight prediction heads are inserted at intermediate layers of the SSL backbone, allowing clients to terminate computation based on local constraints and task requirements. We further introduce a layer-wise, depth-aware partial aggregation strategy to better utilize representations from different network depths. Experiments show that the framework reduces edge overhead, supports heterogeneous hardware, and maintains competitive performance in resource-constrained federated environments.

Adaptive Federated Fine-Tuning of Self-Supervised Speech Representations

Abstract

Integrating Federated Learning (FL) with self-supervised learning (SSL) enables privacy-preserving fine-tuning for speech tasks. However, federated environments exhibit significant heterogeneity: clients differ in computational capacity, causing straggler effects under unified fine-tuning, while diverse downstream tasks require different representation depths, making full-model updates inefficient. To address these challenges, we propose an adaptive federated fine-tuning framework with early exits. Lightweight prediction heads are inserted at intermediate layers of the SSL backbone, allowing clients to terminate computation based on local constraints and task requirements. We further introduce a layer-wise, depth-aware partial aggregation strategy to better utilize representations from different network depths. Experiments show that the framework reduces edge overhead, supports heterogeneous hardware, and maintains competitive performance in resource-constrained federated environments.
Paper Structure (15 sections, 1 equation, 1 figure, 3 tables)

This paper contains 15 sections, 1 equation, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Multi-Exit Elastic Backbone
  4. Resource and Task-Aware Local Training
  5. Layer-wise Depth-aware Partial Aggregation
  6. Experimental Setup
  7. Datasets and Downstream Tasks
  8. Model Architecture and Early-Exit Configuration
  9. Federated Learning Environment and Heterogeneity
  10. Experimental Results
  11. Layer-wise Performance Analysis under Centralized and Federated Settings
  12. Layer-wise Partial Aggregation for Heterogeneous Clients
  13. Memory Cost under Varying Model Depth
  14. Conclusion
  15. Generative AI Use Disclosure

Figures (1)

  • Figure 1: Overview of the proposed Adaptive Federated Learning framework. Heterogeneous clients dynamically select the training depth based on local resources. The server employs a Depth-weighted Layer-wise Partial Aggregation strategy, where deeper layers are updated exclusively by high-resource clients.