Inverse Image-Based Rendering for Light Field Generation from Single Images

TL;DR

This work addresses generating rich 4D light fields from a single image by reframing light-field synthesis as an inverse rendering problem. It introduces iIBRnet, a Transformer-based neural renderer that reconstructs continuous light flows in a 4D ray space, using cross-ray attention and disparity-informed ray encoding, along with occlusion-aware inpainting to maintain consistency across novel views. The method leverages EPIs and Plücker-like ray representations to couple source and target rays, optimized with $L_c$, $L_w$, and $L_{epi}$ losses, and extends from 2D to 4D iIBR with a farthest-view-first rendering strategy. Across four datasets, including synthetic Pov-ray and real-world Stanford/LLFF, iIBR achieves state-of-the-art or strong performance in novel-view synthesis and demonstrates robust zero-shot generalization to real imagery, enabling photo-realistic digital refocusing from a single image.

Abstract

A concept of light-fields computed from multiple view images on regular grids has proven its benefit for scene representations, and supported realistic renderings of novel views and photographic effects such as refocusing and shallow depth of field. In spite of its effectiveness of light flow computations, obtaining light fields requires either computational costs or specialized devices like a bulky camera setup and a specialized microlens array. In an effort to broaden its benefit and applicability, in this paper, we propose a novel view synthesis method for light field generation from only single images, named inverse image-based rendering. Unlike previous attempts to implicitly rebuild 3D geometry or to explicitly represent objective scenes, our method reconstructs light flows in a space from image pixels, which behaves in the opposite way to image-based rendering. To accomplish this, we design a neural rendering pipeline to render a target ray in an arbitrary viewpoint. Our neural renderer first stores the light flow of source rays from the input image, then computes the relationships among them through cross-attention, and finally predicts the color of the target ray based on these relationships. After the rendering pipeline generates the first novel view from a single input image, the generated out-of-view contents are updated to the set of source rays. This procedure is iteratively performed while ensuring the consistent generation of occluded contents. We demonstrate that our inverse image-based rendering works well with various challenging datasets without any retraining or finetuning after once trained on synthetic dataset, and outperforms relevant state-of-the-art novel view synthesis methods.

Figures (10)

• Figure 1: iIBR enables to generate high fidelity and consistent 4D light field from single image
• Figure 2: Difference between IBR and iIBR. IBR renders target image from multiple source image by aggregating correspondence colors. On the contrary, iIBR renders multiple target images from single image by reconstructing unknown light flows, which is shown as lines in the EPI. EPI is 2D slice of 4D light field. A straight line in EPI represents set of correspondence's light flows sampled form different angular resolution.
• Figure 3: Importance of accurate EPI generation. Restoring accurate 2D rays along an EPI in sub-pixel accuracy is importance for novel view synthesis.
• Figure 4: 2D ray generation of iIBR. iIBR restores continuous 2D rays from single pixels, as demonstrated by the angular super-resolution of light field images.
• Figure 5: An overview of iIBRnet. Given a single image, iIBRnet generates novel views. iIBRnet operates in two key stages: (1) the ray transformer, which calculates the relationships between rays from different viewpoints, acting as the mechanism for rendering the light flows of cast rays from input image, and (2) an inpainting process that handles occlusions and updates the input for iIBRnet.