Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedEU: Evidential Uncertainty-Driven Federated Fine-Tuning of Vision Foundation Models for Remote Sensing Image Segmentation

Xiaokang Zhang, Xuran Xiong, Jianzhong Huang, Lefei Zhang

TL;DR

FedEU, a federated optimization framework for fine-tuning RSIS models driven by evidential uncertainty, enables balanced model adaptation across diverse clients by explicitly reducing prediction uncertainty, resulting in more robust and reliable federation outcomes.

Abstract

Remote sensing image segmentation (RSIS) in federated environments has gained increasing attention because it enables collaborative model training across distributed datasets without sharing raw imagery or annotations. Federated RSIS combined with parameter-efficient fine-tuning (PEFT) can unleash the generalization power of pretrained foundation models for real-world applications, with minimal parameter aggregation and communication overhead. However, the dynamic adaptation of pretrained models to heterogeneous client data inevitably increases update uncertainty and compromises the reliability of collaborative optimization due to the lack of uncertainty estimation for each local model. To bridge this gap, we present FedEU, a federated optimization framework for fine-tuning RSIS models driven by evidential uncertainty. Specifically, personalized evidential uncertainty modeling is introduced to quantify epistemic variations of local models and identify high-risk areas under local data distributions. Furthermore, the client-specific feature embedding (CFE) is exploited to enhance channel-aware feature representation while preserving client-specific properties through personalized attention and an element-aware parameter update approach. These uncertainty estimates are uploaded to the server to enable adaptive global aggregation via a Top-k uncertainty-guided weighting (TUW) strategy, which mitigates the impact of distribution shifts and unreliable updates. Extensive experiments on three large-scale heterogeneous datasets demonstrate the superior performance of FedEU. More importantly, FedEU enables balanced model adaptation across diverse clients by explicitly reducing prediction uncertainty, resulting in more robust and reliable federated outcomes. The source codes will be available at https://github.com/zxk688/FedEU.

FedEU: Evidential Uncertainty-Driven Federated Fine-Tuning of Vision Foundation Models for Remote Sensing Image Segmentation

TL;DR

FedEU, a federated optimization framework for fine-tuning RSIS models driven by evidential uncertainty, enables balanced model adaptation across diverse clients by explicitly reducing prediction uncertainty, resulting in more robust and reliable federation outcomes.

Abstract

Remote sensing image segmentation (RSIS) in federated environments has gained increasing attention because it enables collaborative model training across distributed datasets without sharing raw imagery or annotations. Federated RSIS combined with parameter-efficient fine-tuning (PEFT) can unleash the generalization power of pretrained foundation models for real-world applications, with minimal parameter aggregation and communication overhead. However, the dynamic adaptation of pretrained models to heterogeneous client data inevitably increases update uncertainty and compromises the reliability of collaborative optimization due to the lack of uncertainty estimation for each local model. To bridge this gap, we present FedEU, a federated optimization framework for fine-tuning RSIS models driven by evidential uncertainty. Specifically, personalized evidential uncertainty modeling is introduced to quantify epistemic variations of local models and identify high-risk areas under local data distributions. Furthermore, the client-specific feature embedding (CFE) is exploited to enhance channel-aware feature representation while preserving client-specific properties through personalized attention and an element-aware parameter update approach. These uncertainty estimates are uploaded to the server to enable adaptive global aggregation via a Top-k uncertainty-guided weighting (TUW) strategy, which mitigates the impact of distribution shifts and unreliable updates. Extensive experiments on three large-scale heterogeneous datasets demonstrate the superior performance of FedEU. More importantly, FedEU enables balanced model adaptation across diverse clients by explicitly reducing prediction uncertainty, resulting in more robust and reliable federated outcomes. The source codes will be available at https://github.com/zxk688/FedEU.
Paper Structure (28 sections, 14 equations, 8 figures, 5 tables, 1 algorithm)

This paper contains 28 sections, 14 equations, 8 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related works
  3. Federated Image Segmentation
  4. Federated Fine-Tuning
  5. Personalization and Regularization in Federated Optimization
  6. Methodology
  7. Preliminaries
  8. Client-Specific Feature Embeddings
  9. Uncertainty-Guided Local Training
  10. Uncertainty-Guided Global Aggregation
  11. Adaptive Update of CFE
  12. Implementation
  13. Experiments
  14. Datasets
  15. GaoFen-7 (GF-7) Building Dataset
  16. ...and 13 more sections

Figures (8)

  • Figure 1: The framework of FedEU. The local network is a typical encoder-decoder framework with a segmentation head and an evidential uncertainty (EU) head. Furthermore, the CFE module is introduced to enable each client to assign personalized attention to different channels and calibrate its feature representations accordingly through dynamic parameter update. In the local training, the local model generates predictive evidential uncertainties via the EU head. In the global aggregation, the Top-$k$ uncertainty-based weights are used to guide model parameter aggregation on the server side.
  • Figure 2: Illustration of the CFE module, which is inspired by wang2022personalizing. It employs a personalized channel attention mechanism by embedding domain identifiers into the feature representations and three MLP layers are adopted to perform gated channel attention. The layer parameters are updated via an element-wise strategy considering the dynamics of the training process.
  • Figure 3: Visual comparison of building extraction results from various FL algorithms on the GF-7 dataset. (a) Original image. (b) Ground truth. (c)-(h) The segmentation results obtained by (c) FedAvg, (d) FedProx, (e) FedSeg, (f) FedTGP, (g) Per-FedAvg, (h) FedBN and (i) FedEU. True positives, true negatives, false positives and false negatives are denoted by white, black, red, and blue, respectively.
  • Figure 4: Visual comparison of water extraction results from various FL algorithms on the GLH-water dataset. (a) Original image. (b) Ground truth. (c)-(h) The segmentation results obtained by (c) FedAvg, (d) FedProx, (e) FedSeg, (f) FedTGP, (g) Per-FedAvg, (h) FedBN and (i) FedEU. True positives, true negatives, false positives and false negatives are denoted by white, black, red, and blue, respectively.
  • Figure 5: Visual comparisons on the GVLM dataset. (a) Original image. (b) Ground truth. (c)-(h) The segmentation results obtained by (c) FedAvg, (d) FedProx, (e) FedSeg, (f) FedTGP, (g) Per-FedAvg, (h) FedBN and (i) FedEU. True positives, true negatives, false positives and false negatives are denoted by white, black, red, and blue, respectively.
  • ...and 3 more figures