Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Online Adaptation for Deep Stereo

Matteo Poggi, Fabio Tosi

TL;DR

The paper tackles domain shift in real-time stereo by proposing a federated online adaptation framework (FedFULL and FedMAD) that distributes the expensive optimization across a fleet of devices. It introduces MADNet 2, a lightweight backbone that uses all-pairs correlation volumes to expand the effective search range while maintaining efficiency. Experiments across KITTI, DrivingStereo, and DSEC demonstrate that federated adaptation can match or surpass on-device adaptation in accuracy while preserving real-time speeds on constrained hardware, with FedMAD offering substantial reductions in communication. This approach enables scalable, cloud-assisted adaptation for fleets of autonomous systems while keeping per-device latency low, with implications for robust depth sensing in diverse deployment conditions.

Abstract

We introduce a novel approach for adapting deep stereo networks in a collaborative manner. By building over principles of federated learning, we develop a distributed framework allowing for demanding the optimization process to a number of clients deployed in different environments. This makes it possible, for a deep stereo network running on resourced-constrained devices, to capitalize on the adaptation process carried out by other instances of the same architecture, and thus improve its accuracy in challenging environments even when it cannot carry out adaptation on its own. Experimental results show how federated adaptation performs equivalently to on-device adaptation, and even better when dealing with challenging environments.

Federated Online Adaptation for Deep Stereo

TL;DR

The paper tackles domain shift in real-time stereo by proposing a federated online adaptation framework (FedFULL and FedMAD) that distributes the expensive optimization across a fleet of devices. It introduces MADNet 2, a lightweight backbone that uses all-pairs correlation volumes to expand the effective search range while maintaining efficiency. Experiments across KITTI, DrivingStereo, and DSEC demonstrate that federated adaptation can match or surpass on-device adaptation in accuracy while preserving real-time speeds on constrained hardware, with FedMAD offering substantial reductions in communication. This approach enables scalable, cloud-assisted adaptation for fleets of autonomous systems while keeping per-device latency low, with implications for robust depth sensing in diverse deployment conditions.

Abstract

We introduce a novel approach for adapting deep stereo networks in a collaborative manner. By building over principles of federated learning, we develop a distributed framework allowing for demanding the optimization process to a number of clients deployed in different environments. This makes it possible, for a deep stereo network running on resourced-constrained devices, to capitalize on the adaptation process carried out by other instances of the same architecture, and thus improve its accuracy in challenging environments even when it cannot carry out adaptation on its own. Experimental results show how federated adaptation performs equivalently to on-device adaptation, and even better when dealing with challenging environments.
Paper Structure (13 sections, 2 equations, 3 figures, 5 tables, 2 algorithms)

This paper contains 13 sections, 2 equations, 3 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Federated Adaptation for Deep Stereo
  4. Background: Online Adaptation for Stereo
  5. Federated Adaptation Framework
  6. Proposed Backbone: MADNet 2
  7. Experimental Results
  8. Experimental Settings
  9. Datasets
  10. Evaluation on KITTI
  11. Evaluation on DrivingStereo
  12. Evaluation on DSEC
  13. Conclusion

Figures (3)

  • Figure 2: Overview of our federated adaptation framework. On the one hand, active nodes run online adaptation (blue and yellow) and periodically send their updated weights to a central server. On the other, a listening client (green) can benefit from the adaptation process carried out by the active nodes, by receiving aggregated weights updates from the server.
  • Figure 3: Ablation study -- impact of the update frequency (top) and number of agents (bottom) on accuracy. We report D1-all (%) on the KITTI dataset for FedFULL (blue) and FedMAD (green).
  • Figure 4: Ablation study -- impact of the update frequency (top) and number of clients (bottom) on traffic. We report MB/s (top) and MB/updates (bottom) exchanged on the KITTI dataset for FedFULL (blue) and FedMAD (green).