N-Dimensional Gaussians for Fitting of High Dimensional Functions

TL;DR

This work tackles the problem of representing high-dimensional appearance through explicit N-D Gaussian mixtures, addressing the challenges of fitting efficiency and render-time evaluation. It introduces a Cholesky-based N-D Gaussian parameterization, a Locality Sensitive Hashing–inspired high-dimensional culling scheme, and an optimization-controlled refinement strategy with parent–child Gaussian dependencies to adaptively add detail. The approach enables compact, accurate modeling of complex, view- and scene-dependent appearance, demonstrated on global illumination with variability and volumetric radiance fields, and shows favorable comparisons to implicit and hybrid baselines in both quality and efficiency. The resulting framework renders in real time and trains within minutes, offering practical impact for high-dimensional explicit representations in rendering and graphics pipelines where variability across multiple input dimensions is critical.

Abstract

In the wake of many new ML-inspired approaches for reconstructing and representing high-quality 3D content, recent hybrid and explicitly learned representations exhibit promising performance and quality characteristics. However, their scaling to higher dimensions is challenging, e.g. when accounting for dynamic content with respect to additional parameters such as material properties, illumination, or time. In this paper, we tackle these challenges for an explicit representations based on Gaussian mixture models. With our solutions, we arrive at efficient fitting of compact N-dimensional Gaussian mixtures and enable efficient evaluation at render time: For fast fitting and evaluation, we introduce a high-dimensional culling scheme that efficiently bounds N-D Gaussians, inspired by Locality Sensitive Hashing. For adaptive refinement yet compact representation, we introduce a loss-adaptive density control scheme that incrementally guides the use of additional capacity towards missing details. With these tools we can for the first time represent complex appearance that depends on many input dimensions beyond position or viewing angle within a compact, explicit representation optimized in minutes and rendered in milliseconds.

Figures (6)

• Figure 1: Our optimization receives a number of query points q of N dimensionality as input. For these given points we estimate which Gaussians can be discarded safely through our N-Dimensional culling. With the remaining ones we evaluate for each q our Gaussian mixture either in N dimensions for surface radiance fields or by first projecting the Gaussians to 3D. Our optimization converges to high quality while it also controlling the introduction of new Gaussians via our Optimization-Controlled Refinement.
• Figure 2: We visualize the relationship between parent (blue) and child (green) Gaussians for different covariance matrices and means.
• Figure 3: Our refinement scheme allows the optimizer to choose where to introduce new Gaussians. In the figure, at different parts of the training procedure, we show in blue the already existing Gaussians and in green the ones that the optimizer has chosen to utilize. Notice the dependent Gaussians adding details to the chair texture.
• Figure 4: Demonstration of the impact to image quality and inference time for different values of LSH threshold. With red arrows we point out the artifacts that appear when the threshold is set to lower values than our choice of $3\sigma$.
• Figure 5: Qualitative results of our method compared 3DGS and Instant-NGP for two different scenes with complex view dependent effects.