GAURA: Generalizable Approach for Unified Restoration and Rendering of Arbitrary Views

TL;DR

GAURA addresses the challenge of rendering high-fidelity novel views from degraded multi-view inputs by introducing degradation-aware priors into a Generalizable Novel View Synthesis framework. It leverages a Degradation-aware Latent Module (DLM) and an Adaptive Residual Module (ARM) to condition both epipolar feature aggregation and ray-based rendering on the input degradation, enabling zero-shot generalization to new scenes and multiple degradations. The method is trained end-to-end on synthetic degraded-clean pairs and shows state-of-the-art or competitive performance across low-light, dehazing, deraining, and motion blur tasks, with efficient fine-tuning to unseen degradations using small amounts of data. GAURA demonstrates robust 3D restoration and rendering without per-scene optimization, offering practical impact for real-world capture conditions and flexible adaptation to new corruption types.

Abstract

Neural rendering methods can achieve near-photorealistic image synthesis of scenes from posed input images. However, when the images are imperfect, e.g., captured in very low-light conditions, state-of-the-art methods fail to reconstruct high-quality 3D scenes. Recent approaches have tried to address this limitation by modeling various degradation processes in the image formation model; however, this limits them to specific image degradations. In this paper, we propose a generalizable neural rendering method that can perform high-fidelity novel view synthesis under several degradations. Our method, GAURA, is learning-based and does not require any test-time scene-specific optimization. It is trained on a synthetic dataset that includes several degradation types. GAURA outperforms state-of-the-art methods on several benchmarks for low-light enhancement, dehazing, deraining, and on-par for motion deblurring. Further, our model can be efficiently fine-tuned to any new incoming degradation using minimal data. We thus demonstrate adaptation results on two unseen degradations, desnowing and removing defocus blur. Code and video results are available at vinayak-vg.github.io/GAURA.

Figures (11)

• Figure 1: Reconstructing 3D scenes from imperfect image captures is highly desirable, e.g., in low light, but is challenging. To address this, we introduce GAURA, a technique designed to render and restore novel views from degraded input views. Unlike previous attempts at this problem, we demonstrate generalization to different scenes and degradation types. In each example, we visualize the imperfect target capture (on the left) and its corresponding clear rendered view (on the right). Please note that the imperfect view from the target viewing angle is not used as input for our method, and we simply visualize the same for simplicity. GAURA faithfully recovers the underlying 3D scene with high geometric accuracy while still managing to generalize across several degradation types.
• Figure 2: Overview of GAURA: 1) Given multi-view images of a scene captured in imperfect conditions, we first extract deep convolutional features for each input view, 2) Using the source view features, we estimate the target clean rendered view via an extended epipolar-based rendering pipeline conditioned on the input imperfection type, 3) By supervising this pipeline end-to-end on paired synthetic data, our degradation-specific latent codes encode discriminative information about each imperfection type and dynamically adapts the rendering modules based on the input corruption type. Our method can directly generalize to any new scene containing real-world degradations during inference.
• Figure 3: Qualitative results on three restoration tasks: low-light enhancement, motion blur removal, and dehazing on real-world datasets. In the first row, our method successfully recovers the underlying 3D scene from poorly lit images much closer to the ground truth than the baselines. In the second row, our method can reconstruct fine details (woven patterns on the ball) with higher fidelity while in the third row, GAURA successfully removes haze from the synthesized views and can accurately match the colors on the palette more closely to the ground truth color.
• Figure 4: Qualitative results of GAURA on additional restoration tasks. Our pretrained model can remove imperfections caused due to rain particles (row 1). With limited finetuning, GAURA can learn to successfully restore any new degradation type (unseen during training). For representation purposes, we visualize results on snow (row 2) and defocus blur (row 3), and our method outperforms degradation-specific restoration baselines.
• Figure 5: Qualitative results on scenes containing multiple imperfections in the input image captures. Upon interpolating the latent codes corresponding to the individual imperfection types, GAURA can "combine" multiple image formation processes and recover the original clean scene effectively.