Fused-Planes: Improving Planar Representations for Learning Large Sets of 3D Scenes

TL;DR

Fused-Planes tackles the resource bottleneck of learning large sets of 3D scenes by introducing a micro/macro planar decomposition and a 3D-aware latent space, enabling two-stage training over scene subsets. The macro component uses shared base planes to capture common structure across scenes, while the micro component remains per-scene and lightweight. Training begins with a subset jointly with the latent space and base planes, then scales to additional scenes with latent supervision and RGB alignment, achieving state-of-the-art resource efficiency among planar representations while preserving rendering quality. This approach makes multi-scene inverse graphics more practical at scale, with public code and robust ablations supporting its effectiveness.

Abstract

To learn large sets of scenes, Tri-Planes are commonly employed for their planar structure that enables an interoperability with image models, and thus diverse 3D applications. However, this advantage comes at the cost of resource efficiency, as Tri-Planes are not the most computationally efficient option. In this paper, we introduce Fused-Planes, a new planar architecture that improves Tri-Planes resource-efficiency in the framework of learning large sets of scenes, which we call "multi-scene inverse graphics". To learn a large set of scenes, our method divides it into two subsets and operates as follows: (i) we train the first subset of scenes jointly with a compression model, (ii) we use that compression model to learn the remaining scenes. This compression model consists of a 3D-aware latent space in which Fused-Planes are learned, enabling a reduced rendering resolution, and shared structures across scenes that reduce scene representation complexity. Fused-Planes present competitive resource costs in multi-scene inverse graphics, while preserving Tri-Planes rendering quality, and maintaining their widely favored planar structure. Our codebase is publicly available as open-source. Our project page can be found at https://fused-planes.github.io .

Figures (11)

• Figure 1: Overview: NeRF methods for MSIG. Comparison of resource costs and rendering quality across recent works when training a scene. Circle sizes represent the NVS quality. Our method presents the lowest training time and memory footprint among all planar representations, while maintaining a similar rendering quality. Fused-Planes-ULW presents the lowest memory requirement. Additional details can be found in \ref{['sec:evaluations']}.
• Figure 2: Fused-Planes architecture and training framework. We learn a set of Fused-Planes $\mathcal{T} = \{T_i\}$ in the latent space of an autoencoder, denoted by the encoder $E_\phi$ and the decoder $D_\psi$. Hence, Fused-Planes render latent images $\tilde{z}_{i,j}$ with reduced resolution, enabling faster rendering and training. Each Fused-Plane $T_i$ is split into a micro plane $T_i^\mathrm{mic}$ which captures scene specific information, and a macro plane $T_i^\mathrm{mac}$ computed via a weighted summation over $M$ shared base planes $\mathcal{B}$, with weights $W_i$. The shared planes $\mathcal{B}$ capture common structure across scenes. To learn our set of Fused-Planes, we start by training a first subset of micro planes $\mathcal{T}_1^\mathrm{mic}$, their corresponding weights $W_i$ and the base planes $\mathcal{B}$, jointly with the encoder $E_\phi$ and decoder $D_\psi$. Subsequently, we learn the remaining scenes by training the micro planes $\mathcal{T}_2^\mathrm{mic}$ and their corresponding weights $W_i$ while fine-tuning $\mathcal{B}$ and $D_\psi$.
• Figure 3: Qualitative comparison Comparison of NVS quality between Fused-Planes and other many-scene methods, on Basel Faces and four ShapeNet categories. NVS is performed on never-seen test views. Fused-Planes (planar) and SPARF-128 (non-planar) showcase superior rendering quality compared to other methods.
• Figure 4: Scaling the number of scenes. Evolution of the total training time $t_\mathrm{tot}$ (left) and total memory footprint $m_\mathrm{tot}$ (right) when scaling the number of scenes ($N$), for Fused-Planes and other methods. Solid lines depict planar representation, while dashed and dotted lines denote non-planar representation. Fused-Planes presents better scaling in terms of resources cost as compared to other planar representations. For clarity, we present in \ref{['fig:x-memory-costs-zoomed']} a magnified version of the memory cost plot focusing on the range $N \in [0,100]$.
• Figure 5: Memory costs. This figure presents the memory costs depicted in \ref{['fig:cost-plots']} within the range $N \in [0, 100]$.