Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Learning for Large-Scale Scene Modeling with Neural Radiance Fields

Teppei Suzuki

TL;DR

This work tailor the model aggregation pipeline in federated learning for NeRF, thereby allowing local updates of NeRF and proposing global pose alignment to align the noisy global pose of clients before the aggregation step.

Abstract

We envision a system to continuously build and maintain a map based on earth-scale neural radiance fields (NeRF) using data collected from vehicles and drones in a lifelong learning manner. However, existing large-scale modeling by NeRF has problems in terms of scalability and maintainability when modeling earth-scale environments. Therefore, to address these problems, we propose a federated learning pipeline for large-scale modeling with NeRF. We tailor the model aggregation pipeline in federated learning for NeRF, thereby allowing local updates of NeRF. In the aggregation step, the accuracy of the clients' global pose is critical. Thus, we also propose global pose alignment to align the noisy global pose of clients before the aggregation step. In experiments, we show the effectiveness of the proposed pose alignment and the federated learning pipeline on the large-scale scene dataset, Mill19.

Federated Learning for Large-Scale Scene Modeling with Neural Radiance Fields

TL;DR

This work tailor the model aggregation pipeline in federated learning for NeRF, thereby allowing local updates of NeRF and proposing global pose alignment to align the noisy global pose of clients before the aggregation step.

Abstract

We envision a system to continuously build and maintain a map based on earth-scale neural radiance fields (NeRF) using data collected from vehicles and drones in a lifelong learning manner. However, existing large-scale modeling by NeRF has problems in terms of scalability and maintainability when modeling earth-scale environments. Therefore, to address these problems, we propose a federated learning pipeline for large-scale modeling with NeRF. We tailor the model aggregation pipeline in federated learning for NeRF, thereby allowing local updates of NeRF. In the aggregation step, the accuracy of the clients' global pose is critical. Thus, we also propose global pose alignment to align the noisy global pose of clients before the aggregation step. In experiments, we show the effectiveness of the proposed pose alignment and the federated learning pipeline on the large-scale scene dataset, Mill19.
Paper Structure (15 sections, 5 equations, 5 figures, 2 tables)

This paper contains 15 sections, 5 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Radiance Fields
  4. Federated Learning
  5. Federated Learning for NeRF
  6. Problem Definition and Assumptions
  7. Training Local Models
  8. Updating Global Model
  9. Global Pose Alignment
  10. Experiments
  11. Experimental Setup
  12. Effect of Pose Alignment
  13. Comparison with Other Training Protocols
  14. Limitations and Future Work
  15. Conclusion

Figures (5)

  • Figure 1: Overview of the proposed federated learning pipeline for neural radiance fields.
  • Figure 2: Overview of the proposed aggregation step. We first cache outputs of the local model on the grid voxels, $\tilde{V}^i_\sigma$ and $\tilde{V}^i_c$, and then add them to the global voxel grid, $V_\sigma$ and $V_c$.
  • Figure 3: The alignment results for various initial errors. The gray dots denote failure cases that increase the initial errors.
  • Figure 4: From left to right, the ground-truth image of the test view, the image rendered by the local model trained with a sufficiently large number of viewpoints around the test view, and the image rendered by the local model trained with a relatively small number of viewpoints.
  • Figure 5: An image rendered by the client model (left) and an image rendered by the cached voxel grids (right). The lattice pattern composed of high-frequency components is broken.