Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations

Mehdi S. M. Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lucic, Daniel Duckworth, Alexey Dosovitskiy, Jakob Uszkoreit, Thomas Funkhouser, Andrea Tagliasacchi

TL;DR

Scene Representation Transformer (SRT) introduces a geometry-free, transformer-based pipeline for interactive novel view synthesis from a few RGB images, handling posed or unposed inputs. An Encoder Transformer converts a set of image patches into a set-latent scene representation, which a Decoder Transformer renders via ray-conditioned attention, enabling rapid, single-pass view synthesis. Across synthetic and Street View datasets, SRT achieves superior PSNR/SSIM with competitive latency compared to NeRF-based and light-field baselines, and demonstrates robustness to pose noise and unposed imagery. The work also shows potential for downstream tasks like semantic segmentation, highlighting the versatility of the learned scene representation for real-world visualization and analysis.

Abstract

A classical problem in computer vision is to infer a 3D scene representation from few images that can be used to render novel views at interactive rates. Previous work focuses on reconstructing pre-defined 3D representations, e.g. textured meshes, or implicit representations, e.g. radiance fields, and often requires input images with precise camera poses and long processing times for each novel scene. In this work, we propose the Scene Representation Transformer (SRT), a method which processes posed or unposed RGB images of a new area, infers a "set-latent scene representation", and synthesises novel views, all in a single feed-forward pass. To calculate the scene representation, we propose a generalization of the Vision Transformer to sets of images, enabling global information integration, and hence 3D reasoning. An efficient decoder transformer parameterizes the light field by attending into the scene representation to render novel views. Learning is supervised end-to-end by minimizing a novel-view reconstruction error. We show that this method outperforms recent baselines in terms of PSNR and speed on synthetic datasets, including a new dataset created for the paper. Further, we demonstrate that SRT scales to support interactive visualization and semantic segmentation of real-world outdoor environments using Street View imagery.

Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations

TL;DR

Scene Representation Transformer (SRT) introduces a geometry-free, transformer-based pipeline for interactive novel view synthesis from a few RGB images, handling posed or unposed inputs. An Encoder Transformer converts a set of image patches into a set-latent scene representation, which a Decoder Transformer renders via ray-conditioned attention, enabling rapid, single-pass view synthesis. Across synthetic and Street View datasets, SRT achieves superior PSNR/SSIM with competitive latency compared to NeRF-based and light-field baselines, and demonstrates robustness to pose noise and unposed imagery. The work also shows potential for downstream tasks like semantic segmentation, highlighting the versatility of the learned scene representation for real-world visualization and analysis.

Abstract

A classical problem in computer vision is to infer a 3D scene representation from few images that can be used to render novel views at interactive rates. Previous work focuses on reconstructing pre-defined 3D representations, e.g. textured meshes, or implicit representations, e.g. radiance fields, and often requires input images with precise camera poses and long processing times for each novel scene. In this work, we propose the Scene Representation Transformer (SRT), a method which processes posed or unposed RGB images of a new area, infers a "set-latent scene representation", and synthesises novel views, all in a single feed-forward pass. To calculate the scene representation, we propose a generalization of the Vision Transformer to sets of images, enabling global information integration, and hence 3D reasoning. An efficient decoder transformer parameterizes the light field by attending into the scene representation to render novel views. Learning is supervised end-to-end by minimizing a novel-view reconstruction error. We show that this method outperforms recent baselines in terms of PSNR and speed on synthetic datasets, including a new dataset created for the paper. Further, we demonstrate that SRT scales to support interactive visualization and semantic segmentation of real-world outdoor environments using Street View imagery.
Paper Structure (45 sections, 9 equations, 14 figures, 3 tables)

This paper contains 45 sections, 9 equations, 14 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. NeRF with Re-projection
  4. Light Fields
  5. Transformers
  6. Method
  7. Pose information
  8. CNN -- \ref{['fig:architecture']} (left)
  9. Encoder Transformer -- \ref{['fig:architecture']} (center)
  10. Decoder Transformer -- \ref{['fig:architecture']} (right)
  11. Training and Inference
  12. Experimental Results
  13. Datasets
  14. Neural 3D Mesh Renderer Dataset (NMR) kato2018neural
  15. MultiShapeNet (MSN)
  16. ...and 30 more sections

Figures (14)

  • Figure 1: Model overview -- SRT encodes a collection of images into the scene representation: a set of latent features. Novel views are rendered in real-time by attending into the latent representation with light field rays, see \ref{['fig:architecture']} for details.
  • Figure 2: Network architecture -- Given a set of posed (SRT) or unposed (UpSRT) RGB images, a CNN extracts patch features, which are processed by an encoder transformer, leading to the set-latent scene representation. Novel views are then rendered by attending into the scene representation for a given ray pose, from which an image can then be rendered. Details in \ref{['sec:method', 'sec:app:architecture']}.
  • Figure 3: Qualitative NMR results -- While PixelNeRF has high-quality for target views close to the input (middle), results quickly degrade when further away (bottom). The quality of SRT renders is much more consistent and outperforms both baselines. More results are available at https://srt-paper.github.io/.
  • Figure 4: Qualitative results on MultiShapeNet -- LFN does not scale to this demanding dataset due to its global latent conditioning. With perfect input camera poses, PixelNeRF resolves details in the center of the scene more sharply for target views nearby the inputs (middle). This quickly changes for views further away (bottom), where PixelNeRF produces projection artifacts even with perfect pose, while SRT's results are more coherent. PixelNeRF further has trouble compensating for noisy cameras, where SRT only experiences a mild drop in quality. Finally, UpSRT is the only model that can be run without input camera poses at all (see \ref{['sec:exp:sensitivity']}).
  • Figure 5: Attention visualization -- Input patches that the green input patch attends into in the encoder, and the first & second decoder layers attend into when rendering the marked query ray at the intersection of the green lines. The model learns to attend into the same 3D positions, and to store global information into specific tokens (along the bottom edge). The decoder first attends into the global patches, then into relevant 3D positions of the scene.
  • ...and 9 more figures