DNRSelect: Active Best View Selection for Deferred Neural Rendering

TL;DR

DNRSelect tackles the data-hungry nature of Deferred Neural Rendering by integrating a reinforcement learning–driven view selector and a 3D texture aggregator. The RL-based selector identifies informative views using rasterized images, reducing the need for ray-traced data, while the 3D texture aggregator fuses depth, normal, and UV cues to maintain geometric consistency with sparse viewpoints. The framework trains in two stages: first learning a coarse DNR with selected views, then fine-tuning using the chosen ray-traced views and a multi-term loss to sharpen textures. Experiments on NeRF-Synthetic demonstrate strong rendering fidelity with fewer views, and ablations confirm the value of each component, highlighting DNRSelect’s potential for more data-efficient neural rendering in robotics and related applications.

Abstract

Deferred neural rendering (DNR) is an emerging computer graphics pipeline designed for high-fidelity rendering and robotic perception. However, DNR heavily relies on datasets composed of numerous ray-traced images and demands substantial computational resources. It remains under-explored how to reduce the reliance on high-quality ray-traced images while maintaining the rendering fidelity. In this paper, we propose DNRSelect, which integrates a reinforcement learning-based view selector and a 3D texture aggregator for deferred neural rendering. We first propose a novel view selector for deferred neural rendering based on reinforcement learning, which is trained on easily obtained rasterized images to identify the optimal views. By acquiring only a few ray-traced images for these selected views, the selector enables DNR to achieve high-quality rendering. To further enhance spatial awareness and geometric consistency in DNR, we introduce a 3D texture aggregator that fuses pyramid features from depth maps and normal maps with UV maps. Given that acquiring ray-traced images is more time-consuming than generating rasterized images, DNRSelect minimizes the need for ray-traced data by using only a few selected views while still achieving high-fidelity rendering results. We conduct detailed experiments and ablation studies on the NeRF-Synthetic dataset to demonstrate the effectiveness of DNRSelect. The code will be released.

Figures (8)

• Figure 1: DNRSelect: We employ reinforcement learning to select a sequence of views that maximizes information gain, enhancing the quality of novel view synthesis with minimal additional resources, thereby aiding DNR in reconstruction.
• Figure 2: The overall pipeline of DNRSelect mainly includes two steps. In Step 1, we introduce a reinforcement learning-based (RL-based) view selector to minimize the reliance on abundant ray-traced images. Meanwhile, the DNR model learns the coarse geometric priors of the target object. In Step 2, the coarse DNR model is fine-tuned on the selected ray-traced images to achieve finer textures. Additionally, we enhance the hierarchical sampling in the vanilla DNR with a novel 3D texture aggregator, which accounts for the geometric information missing due to sparse selective views.
• Figure 3: Illustration of the proposed RL-based view selector. The selector encodes the selected views into embeddings and utilizes a convolutional neural network to transform the neural textures into features representing the current state. Then, Q-table is employed to approximate the mapping between the current states and their corresponding values. After optimization, the next best view is selected using a greedy algorithm that maximizes the expected reward.
• Figure 4: A selection of our preprocessed NeRF-Synthetics dataset is illustrated in the figure.
• Figure 5: Visualization of the rendering details for our method at 25 and 100 view numbers, compared to other methods at 25 view numbers, alongside the ground truth.