Pay Attention to Where You Look

TL;DR

The paper tackles the problem of equal treatment of multiple source views in few-shot novel view synthesis by introducing a camera-weighting mechanism that assigns per-view relevance to the target. It presents two branches: deterministic weighting (distance, angle, and kernel-based cues) and an attention-based approach (pose embeddings with cross-attention). The methods are compatible with existing NeRF-style pipelines (e.g., PixelNeRF, GeNVS), with deterministic weighting offering plug-in simplicity and attention-based weighting requiring targeted training of the weighting module. Experimental results on SRN Cars and SRN Multi-Chairs show that adaptive weighting improves rendering quality and robustness, particularly when a near-target view is available and as more input views are added, illustrating practical gains for real-world NVS applications.

Abstract

Novel view synthesis (NVS) has advanced with generative modeling, enabling photorealistic image generation. In few-shot NVS, where only a few input views are available, existing methods often assume equal importance for all input views relative to the target, leading to suboptimal results. We address this limitation by introducing a camera-weighting mechanism that adjusts the importance of source views based on their relevance to the target. We propose two approaches: a deterministic weighting scheme leveraging geometric properties like Euclidean distance and angular differences, and a cross-attention-based learning scheme that optimizes view weighting. Additionally, models can be further trained with our camera-weighting scheme to refine their understanding of view relevance and enhance synthesis quality. This mechanism is adaptable and can be integrated into various NVS algorithms, improving their ability to synthesize high-quality novel views. Our results demonstrate that adaptive view weighting enhances accuracy and realism, offering a promising direction for improving NVS.

Figures (7)

• Figure 1: Example of the source camera weighting problem. (a) shows equal weighting of camera views even though some views capture non-useful information, and (b) shows prioritized weighting on the darker source camera as it is closest to the target view.
• Figure 2: Process for converting a camera pose matrix to a pose embedding vector.
• Figure 3: Process of the cross-attention weighting algorithm.
• Figure 4: Weighting sustains growth on PixelNeRF. While baseline (mean) performance plateaus with increasing input views, error weighting sustains the growth. Each method number of input views pair is evaluated on the entire SRN Cars test distribution with 3 random target views for each scene tested.
• Figure 5: Weighting sustains growth on GeNVS. While baseline (mean) performance plateaus with increasing input views, error weighting and cross-attention weighting sustains the growth. Evaluation setup is the same as the one used in Figure \ref{['fig:num_input_views_pixelnerf']}.