WateRF: Robust Watermarks in Radiance Fields for Protection of Copyrights

TL;DR

An innovative watermarking method that can be employed in both representations of NeRF is introduced by fine-tuning NeRF to embed binary messages in the rendering process and achieves state-of-the-art performance with faster training speed over the compared state-of-the-art methods.

Abstract

The advances in the Neural Radiance Fields (NeRF) research offer extensive applications in diverse domains, but protecting their copyrights has not yet been researched in depth. Recently, NeRF watermarking has been considered one of the pivotal solutions for safely deploying NeRF-based 3D representations. However, existing methods are designed to apply only to implicit or explicit NeRF representations. In this work, we introduce an innovative watermarking method that can be employed in both representations of NeRF. This is achieved by fine-tuning NeRF to embed binary messages in the rendering process. In detail, we propose utilizing the discrete wavelet transform in the NeRF space for watermarking. Furthermore, we adopt a deferred back-propagation technique and introduce a combination with the patch-wise loss to improve rendering quality and bit accuracy with minimum trade-offs. We evaluate our method in three different aspects: capacity, invisibility, and robustness of the embedded watermarks in the 2D-rendered images. Our method achieves state-of-the-art performance with faster training speed over the compared state-of-the-art methods.

Figures (7)

• Figure 1: The NeRF model can be fine-tuned to embed the watermark into the images of the novel view. The figure above shows the model owner Alice and her NeRF model is stolen by Bob. Alice can identify the copyright by using her watermark decoder. Our method embeds the watermark into all of the rendered images.
• Figure 2: WateRF overview. Phase 1: We train the encoder and the decoder to extract messages. After phase 1, we do not use the encoder. Phase 2: We fine-tune the NeRF to embed the messages into the rendered images. (a) We disable auto-differentiation and render a full-resolution image to save memory. (b) We use DWT for the rendered images and choose the LL subband as the input of the pre-trained decoder. (c) We enable auto-differentiation and render the images patch by patch. Then the NeRF is optimized using Eq \ref{['eq:loss_full']} and Eq \ref{['eq:patch_loss']}.
• Figure 3: Identification results. We average the bit accuracy for each 500 messages. Our method not only works efficiently with just one key but also works well with arbitrarily generated messages. We show the results on $M_L$ = 16 bits.
• Figure 4: Qualitative comparisons We show the differences (×10) between the rendered images and the ground truth. Our method achieves higher PSNR and bit accuracy than CopyRNeRF.
• Figure 5: Reconstruction quality comparisons We evaluate our full method, our method without patch loss and our method without frequency domain about 16 bits with NeRF mildenhall2021nerf.