GOEmbed: Gradient Origin Embeddings for Representation Agnostic 3D Feature Learning

TL;DR

GOEmbed addresses the challenge of encoding information from 2D views into diverse 3D representations without relying on large pretrained image feature extractors. It defines a representation-agnostic embedding by taking the gradient of the rendering error with respect to a zero-origin 3D representation, enabling application to MLPs, Triplanes, voxel grids, and beyond. The method is extended to diffusion-based generation via GOEmbedFusion, which achieves state-of-the-art $\text{FID}=22.12$ and $\text{KID}=0.6$ on OmniObject3D and strengthens sparse-view reconstruction, illustrating practical gains in 3D synthesis and reconstruction. Overall, GOEmbed offers a lightweight, broadly compatible mechanism to infuse 2D information into 3D representations, reducing dependence on pretrained 2D backbones and expanding applicability across current and future 3D representations.$

Abstract

Encoding information from 2D views of an object into a 3D representation is crucial for generalized 3D feature extraction. Such features can then enable 3D reconstruction, 3D generation, and other applications. We propose GOEmbed (Gradient Origin Embeddings) that encodes input 2D images into any 3D representation, without requiring a pre-trained image feature extractor; unlike typical prior approaches in which input images are either encoded using 2D features extracted from large pre-trained models, or customized features are designed to handle different 3D representations; or worse, encoders may not yet be available for specialized 3D neural representations such as MLPs and hash-grids. We extensively evaluate our proposed GOEmbed under different experimental settings on the OmniObject3D benchmark. First, we evaluate how well the mechanism compares against prior encoding mechanisms on multiple 3D representations using an illustrative experiment called Plenoptic-Encoding. Second, the efficacy of the GOEmbed mechanism is further demonstrated by achieving a new SOTA FID of 22.12 on the OmniObject3D generation task using a combination of GOEmbed and DFM (Diffusion with Forward Models), which we call GOEmbedFusion. Finally, we evaluate how the GOEmbed mechanism bolsters sparse-view 3D reconstruction pipelines.

Figures (7)

• Figure 1: We propose the GOEmbed (Gradient Origin Embedding) mechanism that encodes source views ($o^\text{ctxt}$) and camera parameters ($\phi^\text{ctxt}$) into arbitrary 3D Radiance-Field representations $g(c,d)$ (sec. \ref{['sec:method']}). We show how these general-purpose GOEmbeddings can be used in the context of 3D DFMs (Diffusion with Forward Models) (sec. \ref{['sec:3d_gen']}) and for sparse-view 3D reconstruction (sec. \ref{['sec:3d_recon']}).
• Figure 2: GOEmbed illustration. We demonstrate the mechanism here using the Triplane representation for $g(c, d)$, but note that this can be applied to other representations as well. The GOEmbed mechanism (eq. \ref{['eq:goembed_enc']}) consists of two steps. First we render the origin $\zeta_0$ from the context-poses $\phi^\text{ctxt}$; then we compute the gradient of the MSE between the renders and the source-views $o^\text{ctxt}$ wrt. the origin $\zeta_0$ which gives us the GOEmbed encoding $\zeta_\text{enc}$.
• Figure 2: Illustration diagram for the GOEmbed Plenoptic Encoding experimental setup.
• Figure 3: Plenoptic Encoding Qualitative Evaluation. The rows MLP, Triplane and Voxel-grid show the renders of the GOEmbed encoded representations from the target-view respectively, depicting the varying amounts with which GOEmbed is capable of encoding the 2D source information into 3D. The colour-coded columns demonstrate the effect of varying the number of source-views (top row) used in the GOEmbed encoding, viz. 1, 2, 3, and 4. The SSO column shows the target render of the single-scene-overfitted representation while the G.T. column shows the mesh-render from the dataset (repeated for clarity). The rightmost column visualizes the pixelwise squared error between the G.T. and the SSO.
• Figure 3: Illustration diagram for the GOEmbed 3D reconstruction experimental setup.