$S^2$NeRF: Privacy-preserving Training Framework for NeRF

TL;DR

This work addresses privacy risks in NeRF training by introducing SplitNeRF, a split-learning framework for collaborative NeRF training that keeps private scene data on the client. It reveals two gradient-based attacks—Surrogate Model Attack and Scene-aided Surrogate Model Attack—that can reconstruct private scene information from exchanged gradients and limited leaks. To remediate these vulnerabilities, it proposes $S^2$NeRF, a defense that adds decaying gradient noise proportional to gradient norms, achieving strong privacy protection with controlled loss of NeRF utility. The approach is validated on three indoor NeRF datasets, demonstrating robust defense performance and practical applicability, including a light version suitable for resource-constrained deployments.

Abstract

Neural Radiance Fields (NeRF) have revolutionized 3D computer vision and graphics, facilitating novel view synthesis and influencing sectors like extended reality and e-commerce. However, NeRF's dependence on extensive data collection, including sensitive scene image data, introduces significant privacy risks when users upload this data for model training. To address this concern, we first propose SplitNeRF, a training framework that incorporates split learning (SL) techniques to enable privacy-preserving collaborative model training between clients and servers without sharing local data. Despite its benefits, we identify vulnerabilities in SplitNeRF by developing two attack methods, Surrogate Model Attack and Scene-aided Surrogate Model Attack, which exploit the shared gradient data and a few leaked scene images to reconstruct private scene information. To counter these threats, we introduce $S^2$NeRF, secure SplitNeRF that integrates effective defense mechanisms. By introducing decaying noise related to the gradient norm into the shared gradient information, $S^2$NeRF preserves privacy while maintaining a high utility of the NeRF model. Our extensive evaluations across multiple datasets demonstrate the effectiveness of $S^2$NeRF against privacy breaches, confirming its viability for secure NeRF training in sensitive applications.

Figures (17)

• Figure 1: Overview of SplitNeRF framework. Traditional NeRF training requires the uploading of all training data, including scene images and corresponding camera poses, to a central server, presenting a substantial risk to privacy. We propose a split learning-based NeRF training framework, named SplitNeRF. The entire NeRF training model is divided into two parts, the server model and the client model. Specifically, clients are required only to send camera poses to the server, while keeping private image data local. During training, the client transmits a series of poses to the server, which then calculates and sends back the embeddings of these poses. The client proceeds to compute the colors and densities associated with these poses, followed by rendering and loss function calculations. The client then sends the gradients back to the server, thus completing the backpropagation process.
• Figure 2: Overview of Surrogate Model Attack against SplitNeRF. The attacker(curious server) attempts to mimic the representation capabilities of the client model to generate a 3D description of the client's private scenario. To achieve this, the attacker sets up a surrogate model along with dummy image data. The attacker utilizes backpropagation to optimize both the surrogate model and dummy image with the surrogate model loss $L_{surr}$, which includes gradient distance loss $L_g$ and dummy reconstruction loss $L_{dummy}$.
• Figure 3: Surrogate Model Attack results on the three datasets, utilizing $\frac{10}{t}$ learning rate decay schemes, where $t$ denotes the current learning epoch index. The attacker successfully restores partial outlines of the actual scenes across all datasets. Even though the views generated by the attacker are rendered in grayscale, it remains possible to distinguish indoor items and layout scenarios. The level of detail in the reconstructed scenes highlights a significant privacy violation, as it exposes sensitive information about the structure and contents of the scenes.
• Figure 4: Scene-aided Surrogate Model Attack Results. The results show the attack's ability to significantly restore high-quality, colored views from just one leaked picture, closely resembling the original scene. The fidelity of the attack extends beyond the specific pose of the leaked image, accurately replicating various other poses and effectively capturing the entire scene.
• Figure 5: The defense results of $S^2$NeRF with the configure $c = 1.2, r = 0.0001$ under $\frac{10}{t}$-attack, where $c$ and $r$ denote the noise scale and noise decay ratio. $S^2$NeRF achieves a balanced trade-off between privacy protection and model utility. In particular, the attacker can not recover any useful information, while the NeRF with defense can still reflect the appearance of the scene, albeit with some noise in the views.