Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Neural Radiance Field for Distributed Intelligence

Yintian Zhang, Ziyu Shao

TL;DR

The paper tackles privacy constraints in NeRF training when image data is distributed across owners. It proposes FedNeRF, an FL-based approach that trains a global NeRF without transferring raw images, and validates the concept on a resource-rich FL testbed. A channel-aware client selection method, FedNeRF-CS, uses a hyper-parameter $Q$ to balance downlink transmission rate and rendering quality (PSNR) in wireless settings, demonstrated with real devices over multiple rounds. The results demonstrate feasible privacy-preserving NeRF training in distributed data storage contexts and provide a practical platform for deploying NeRF in federated, heterogeneous networks.

Abstract

Novel view synthesis (NVS) is an important technology for many AR and VR applications. The recently proposed Neural Radiance Field (NeRF) approach has demonstrated superior performance on NVS tasks, and has been applied to other related fields. However, certain application scenarios with distributed data storage may pose challenges on acquiring training images for the NeRF approach, due to strict regulations and privacy concerns. In order to overcome this challenge, we focus on FedNeRF, a federated learning (FL) based NeRF approach that utilizes images available at different data owners while preserving data privacy. In this paper, we first construct a resource-rich and functionally diverse federated learning testbed. Then, we deploy FedNeRF algorithm in such a practical FL system, and conduct FedNeRF experiments with partial client selection. It is expected that the studies of the FedNeRF approach presented in this paper will be helpful to facilitate future applications of NeRF approach in distributed data storage scenarios.

Federated Neural Radiance Field for Distributed Intelligence

TL;DR

The paper tackles privacy constraints in NeRF training when image data is distributed across owners. It proposes FedNeRF, an FL-based approach that trains a global NeRF without transferring raw images, and validates the concept on a resource-rich FL testbed. A channel-aware client selection method, FedNeRF-CS, uses a hyper-parameter to balance downlink transmission rate and rendering quality (PSNR) in wireless settings, demonstrated with real devices over multiple rounds. The results demonstrate feasible privacy-preserving NeRF training in distributed data storage contexts and provide a practical platform for deploying NeRF in federated, heterogeneous networks.

Abstract

Novel view synthesis (NVS) is an important technology for many AR and VR applications. The recently proposed Neural Radiance Field (NeRF) approach has demonstrated superior performance on NVS tasks, and has been applied to other related fields. However, certain application scenarios with distributed data storage may pose challenges on acquiring training images for the NeRF approach, due to strict regulations and privacy concerns. In order to overcome this challenge, we focus on FedNeRF, a federated learning (FL) based NeRF approach that utilizes images available at different data owners while preserving data privacy. In this paper, we first construct a resource-rich and functionally diverse federated learning testbed. Then, we deploy FedNeRF algorithm in such a practical FL system, and conduct FedNeRF experiments with partial client selection. It is expected that the studies of the FedNeRF approach presented in this paper will be helpful to facilitate future applications of NeRF approach in distributed data storage scenarios.
Paper Structure (5 sections, 4 figures, 3 tables)

This paper contains 5 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. The Federated Learning Testbed
  3. Conducting FedNeRF Tasks in the FL Testbed
  4. Performance of FedNeRF with Client Selection in FL Testbed
  5. Conclusion

Figures (4)

  • Figure 1: An overview of the proposed FedNeRF approach. Through several rounds of communication between the clients and the FL server, the collaboratively trained global NeRF model can learn omnidirectional scene information in a data privacy-preserving manner.
  • Figure 2: The structure of the federated learning testbed
  • Figure 3: The implementation of FedNeRF in the FL testbed
  • Figure 4: Experiment results of the FedNeRF-CS on real wireless devices