Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedRS-Bench: Realistic Federated Learning Datasets and Benchmarks in Remote Sensing

Haodong Zhao, Peng Peng, Chiyu Chen, Linqing Huang, Gongshen Liu

TL;DR

This paper tackles the lack of realistic federated learning benchmarks for remote sensing by introducing FedRS, a dataset built from eight real RS sources into 135 single-source clients, and FedRS-Bench, a comprehensive evaluation suite with ten FL algorithms evaluated on CNN and ResNet18 backbones. FedRS unifies 64×64 images into 15 semantic categories and includes a simpler FedRS-5 subset to probe scalability and heterogeneity, with two partition schemes ($NIID$-1 and $NIID$-2) and partial participation (10 clients per round across 500 rounds). The study finds that federated learning generally improves over isolated training, but no single FL method dominates across all realistic scenarios, and privacy-preserving approaches like FedDP incur accuracy costs; results also highlight the importance of model architecture, data heterogeneity, and the benefits of pre-trained models with partial updates. By releasing FedRS and FedRS-Bench publicly, the work provides a standard, challenging testbed to drive fair comparisons and future advances in large-scale RS FL, with potential extensions toward personalization, domain adaptation, and multimodal data.

Abstract

Remote sensing (RS) images are usually produced at an unprecedented scale, yet they are geographically and institutionally distributed, making centralized model training challenging due to data-sharing restrictions and privacy concerns. Federated learning (FL) offers a solution by enabling collaborative model training across decentralized RS data sources without exposing raw data. However, there lacks a realistic federated dataset and benchmark in RS. Prior works typically rely on manually partitioned single dataset, which fail to capture the heterogeneity and scale of real-world RS data, and often use inconsistent experimental setups, hindering fair comparison. To address this gap, we propose a realistic federated RS dataset, termed FedRS. FedRS consists of eight datasets that cover various sensors and resolutions and builds 135 clients, which is representative of realistic operational scenarios. Data for each client come from the same source, exhibiting authentic federated properties such as skewed label distributions, imbalanced client data volumes, and domain heterogeneity across clients. These characteristics reflect practical challenges in federated RS and support evaluation of FL methods at scale. Based on FedRS, we implement 10 baseline FL algorithms and evaluation metrics to construct the comprehensive FedRS-Bench. The experimental results demonstrate that FL can consistently improve model performance over training on isolated data silos, while revealing performance trade-offs of different methods under varying client heterogeneity and availability conditions. We hope FedRS-Bench will accelerate research on large-scale, realistic FL in RS by providing a standardized, rich testbed and facilitating fair comparisons across future works. The source codes and dataset are available at https://fedrs-bench.github.io/.

FedRS-Bench: Realistic Federated Learning Datasets and Benchmarks in Remote Sensing

TL;DR

This paper tackles the lack of realistic federated learning benchmarks for remote sensing by introducing FedRS, a dataset built from eight real RS sources into 135 single-source clients, and FedRS-Bench, a comprehensive evaluation suite with ten FL algorithms evaluated on CNN and ResNet18 backbones. FedRS unifies 64×64 images into 15 semantic categories and includes a simpler FedRS-5 subset to probe scalability and heterogeneity, with two partition schemes (-1 and -2) and partial participation (10 clients per round across 500 rounds). The study finds that federated learning generally improves over isolated training, but no single FL method dominates across all realistic scenarios, and privacy-preserving approaches like FedDP incur accuracy costs; results also highlight the importance of model architecture, data heterogeneity, and the benefits of pre-trained models with partial updates. By releasing FedRS and FedRS-Bench publicly, the work provides a standard, challenging testbed to drive fair comparisons and future advances in large-scale RS FL, with potential extensions toward personalization, domain adaptation, and multimodal data.

Abstract

Remote sensing (RS) images are usually produced at an unprecedented scale, yet they are geographically and institutionally distributed, making centralized model training challenging due to data-sharing restrictions and privacy concerns. Federated learning (FL) offers a solution by enabling collaborative model training across decentralized RS data sources without exposing raw data. However, there lacks a realistic federated dataset and benchmark in RS. Prior works typically rely on manually partitioned single dataset, which fail to capture the heterogeneity and scale of real-world RS data, and often use inconsistent experimental setups, hindering fair comparison. To address this gap, we propose a realistic federated RS dataset, termed FedRS. FedRS consists of eight datasets that cover various sensors and resolutions and builds 135 clients, which is representative of realistic operational scenarios. Data for each client come from the same source, exhibiting authentic federated properties such as skewed label distributions, imbalanced client data volumes, and domain heterogeneity across clients. These characteristics reflect practical challenges in federated RS and support evaluation of FL methods at scale. Based on FedRS, we implement 10 baseline FL algorithms and evaluation metrics to construct the comprehensive FedRS-Bench. The experimental results demonstrate that FL can consistently improve model performance over training on isolated data silos, while revealing performance trade-offs of different methods under varying client heterogeneity and availability conditions. We hope FedRS-Bench will accelerate research on large-scale, realistic FL in RS by providing a standardized, rich testbed and facilitating fair comparisons across future works. The source codes and dataset are available at https://fedrs-bench.github.io/.
Paper Structure (18 sections, 12 figures, 4 tables)

This paper contains 18 sections, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Descriptions of FedRS
  4. Constituent Datasets and Properties
  5. Federated Training Data Partitioning (135 Clients)
  6. Training Datasets Statistics
  7. Experiments
  8. Experimental Setup
  9. FedRS-Bench Results
  10. Further Exploration
  11. Conclusion
  12. Limitations
  13. Dataset Statistics
  14. Data sources
  15. Benchmark Results
  16. ...and 3 more sections

Figures (12)

  • Figure 1: Samples of FedRS.
  • Figure 2: Visualization of features of from different sources using t-SNE. The corresponding categories and silhouette scores shahapure2020cluster after dimensionality reduction are shown in the subtitles, reflecting the intra-category feature deviation from different sources. Please refer to Figure \ref{['fig:tsne_sources_all']} for all results.
  • Figure 3: Data distribution between different clients under NIID-1 partition. (a) Category distribution, each client only has a part of the 15 categories; (b) Source distribution, each client's data comes from only one source, and the number of clients contributed by each source varies greatly.
  • Figure 4: Test accuracy curve of FL baseline method during the communication round of training. More results on $T_B$ and FedRS-5 are shown in Appendix \ref{['sec:curves']}.
  • Figure 5: Taking the Residential class as an example, there are obvious differences in the style of data in different data sources.
  • ...and 7 more figures