Deep Learning on Object-centric 3D Neural Fields

TL;DR

The paper tackles the challenge of performing downstream tasks directly on neural fields (NFs) rather than their discrete reconstructions. It introduces nf2vec, a weight-space encoder that maps an input NF to a compact latent embedding in a single pass, enabling standard deep learning pipelines to operate on NFs alone. The approach is demonstrated across geometry-only fields (SDF/UDF/OF) and NeRFs, enabling tasks such as retrieval, classification, segmentation, completion, and generation with competitive performance and ultra-fast inference. Key contributions include the first NF classification benchmark, analysis of latent-space properties, and demonstrations of latent-space interpolation and cross-modal mappings between NF types, all while highlighting the practical potential of treating NFs as a unified 3D representation. The work paves the way for storage and processing of 3D data as neural fields in real-world pipelines without reconstructing discrete signals.

Abstract

In recent years, Neural Fields (NFs) have emerged as an effective tool for encoding diverse continuous signals such as images, videos, audio, and 3D shapes. When applied to 3D data, NFs offer a solution to the fragmentation and limitations associated with prevalent discrete representations. However, given that NFs are essentially neural networks, it remains unclear whether and how they can be seamlessly integrated into deep learning pipelines for solving downstream tasks. This paper addresses this research problem and introduces nf2vec, a framework capable of generating a compact latent representation for an input NF in a single inference pass. We demonstrate that nf2vec effectively embeds 3D objects represented by the input NFs and showcase how the resulting embeddings can be employed in deep learning pipelines to successfully address various tasks, all while processing exclusively NFs. We test this framework on several NFs used to represent 3D surfaces, such as unsigned/signed distance and occupancy fields. Moreover, we demonstrate the effectiveness of our approach with more complex NFs that encompass both geometry and appearance of 3D objects such as neural radiance fields.

Figures (25)

• Figure 1: Overview of our framework.Left: NFs hold the potential to provide a unified representation of the 3D world. Center: Our framework, dubbed nf2vec, produces a compact representation for an input NF by looking only at its weights. Right: nf2vec embeddings can be used with standard deep-learning machinery to solve various downstream tasks.
• Figure 2: Encoder architecture.Left: Given a NF, we stack its weights and biases to form a matrix $\mathbf{P}$. Right: The nf2vec encoder is a series of linear layers with batch-norms and ReLU activation functions. It processes each row of $\mathbf{P}$ independently and then aggregates all rows of one NF with a max-pooling to produce a compact embedding employed for downstream tasks.
• Figure 3: Training and inference of nf2vec.Left: A Neural Field (NF) represents a 3D object. The NF is composed of an MLP that parametrizes a field function $f$. Center:nf2vec encoder is trained together with an implicit decoder. The implicit decoder processes the embedding produced by the encoder to estimate field values $\hat{f}$. We train the framework similarly to the input NF. Right: At inference time, the learned encoder can be used to obtain a compact embedding from unseen NFs.
• Figure 4: nf2vec reconstructions.
• Figure 5: nf2vec reconstructions.