General Neural Gauge Fields

TL;DR

The paper introduces Neural Gauge Fields, a unified framework to learn gauge transformations alongside neural fields for 3D scene representations. It derives a principled information-based regularization (InfoReg) to counter gauge collapse and proposes an information-invariant gauge transformation (InfoInv) that preserves scene information inherently, reducing computational overhead. The approach supports both continuous (e.g., 3D space to 2D plane) and discrete (e.g., codebook) gauge mappings, and demonstrates superior performance over predefined gauges and prior regularizers in novel-view synthesis and gauge utilization. Together, these methods enable more flexible, efficient, and high-quality neural field representations with broader applicability to texture editing and related tasks.

Abstract

The recent advance of neural fields, such as neural radiance fields, has significantly pushed the boundary of scene representation learning. Aiming to boost the computation efficiency and rendering quality of 3D scenes, a popular line of research maps the 3D coordinate system to another measuring system, e.g., 2D manifolds and hash tables, for modeling neural fields. The conversion of coordinate systems can be typically dubbed as \emph{gauge transformation}, which is usually a pre-defined mapping function, e.g., orthogonal projection or spatial hash function. This begs a question: can we directly learn a desired gauge transformation along with the neural field in an end-to-end manner? In this work, we extend this problem to a general paradigm with a taxonomy of discrete \& continuous cases, and develop a learning framework to jointly optimize gauge transformations and neural fields. To counter the problem that the learning of gauge transformations can collapse easily, we derive a general regularization mechanism from the principle of information conservation during the gauge transformation. To circumvent the high computation cost in gauge learning with regularization, we directly derive an information-invariant gauge transformation which allows to preserve scene information inherently and yield superior performance. Project: https://fnzhan.com/Neural-Gauge-Fields

Figures (13)

• Figure 1: Conceptual illustration of a gauge transformation from 3D point coordinates to a 2D plane. Instead of naively employ a pre-defined orthogonal mapping which incurs overlap on the 2D plane, the proposed neural gauge fields aim to learn the mapping along with neural fields driven by multi-view synthesis loss.
• Figure 2: Continuous gauge transformation from 3D scene to 2D square. Without regularization, the learned gauge transformation will collapse to a small region in the 2D square. Including cycle regularization xiang2021neutex or structural regularization (StruReg) tretschk2021non don't fully solve the problem, and our InfoReg achieves a clearly better regularization performance.
• Figure 3: Discrete gauge transformation from 3D scene to 256 discrete vectors. Without regularization, the learned gauge transformation will collapse to a small number of vector indices (horizontal axis). The learning with cycle regularization xiang2021neutex is still significantly collapse-prone, while our InfoReg enables to alleviates the collapse substantially.
• Figure 4: (a) Earth Mover's distance to regularize the discrepancy between $\overline{p}(y|x)$ and $\overline{q}(y)$ in continuous case (2D plane), where $\overline{p}(y|x)$ and $\overline{q}(y)$ are predicted 2D points within a batch and uniformly sampled points in the 2D plane, respectively. (b) KL divergence to regularize the distribution discrepancy in discrete cases ($N$ basis vectors), where $\overline{p}(y|x)$ and $q(y)$ are the average of predicted one-hot distributions and a discrete uniform distribution, respectively.
• Figure 5: Qualitative results of continuous gauge transformation from 3D space to 2D plane on (a) 360$^{\circ}$ view scenes and (b) limited view scenes.

Theorems & Definitions (1)

• Lemma 1