BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images

TL;DR

BlockGAN tackles the problem of learning controllable, object-aware 3D scene representations from unlabelled 2D images. It introduces a per-object 3D feature generator and a scene composer that aggregate into a unified 3D scene, followed by a perspective rendering module that yields 2D images. The approach enables explicit manipulation of each object's pose and identity, supports adding or removing objects at test time, and captures realistic lighting and shadows. Empirically, BlockGAN achieves competitive image fidelity while delivering strong disentanglement and editing capabilities, illustrating the value of 3D object-centric representations for unsupervised scene understanding and generation.

Abstract

We present BlockGAN, an image generative model that learns object-aware 3D scene representations directly from unlabelled 2D images. Current work on scene representation learning either ignores scene background or treats the whole scene as one object. Meanwhile, work that considers scene compositionality treats scene objects only as image patches or 2D layers with alpha maps. Inspired by the computer graphics pipeline, we design BlockGAN to learn to first generate 3D features of background and foreground objects, then combine them into 3D features for the wholes cene, and finally render them into realistic images. This allows BlockGAN to reason over occlusion and interaction between objects' appearance, such as shadow and lighting, and provides control over each object's 3D pose and identity, while maintaining image realism. BlockGAN is trained end-to-end, using only unlabelled single images, without the need for 3D geometry, pose labels, object masks, or multiple views of the same scene. Our experiments show that using explicit 3D features to represent objects allows BlockGAN to learn disentangled representations both in terms of objects (foreground and background) and their properties (pose and identity).

Figures (18)

• Figure 1: BlockGAN's generator network. Each noise vector $\mathbf{z}_i$ is mapped to deep 3D object features, which are transformed to the desired 3D pose $\boldsymbol{\theta}_i$. Object features are combined into 3D scene features, where the camera pose $\boldsymbol{\theta}_\text{cam}$ is applied, before projection to 2D features that produce the image $\mathbf{x}$.
• Figure 2: BlockGAN's object generator. Each object starts with a constant tensor that is learnt with the rest of the network.
• Figure 3: Left: The camera's viewing volume (frustum) overlaid on scene-space features. We trilinearly resample the scene features based on the viewing volume at the orange dots. Right: The resulting camera-space features before projection to 2D.
• Figure 4: BlockGAN enables explicit spatial manipulation of individual objects (rotation, translation) and changing the identity of background or foreground objects across different datasets: (a) Synth-Car1, (b) Synth-Chair1, (c) Synth-Car2, (d) Synth-Car3, (e) CLEVR2 and (f) CLEVR3. Notice how the shadows and highlights change as objects move around in the scene, and how changing the background lighting affects the appearance of foreground objects. Figure \ref{['fig:linear_interp']} shows similar results on natural images. Please refer to the supplemental material for animated results.
• Figure 5: Removing/adding objects. The red box shows the original scene.