DDMI: Domain-Agnostic Latent Diffusion Models for Synthesizing High-Quality Implicit Neural Representations

TL;DR

A Domain-agnostic Latent Diffusion Model for INRs (DDMI) is proposed that generates adaptive positional embeddings instead of neural networks' weights and a Discrete-to-continuous space Variational AutoEncoder (D2C-VAE), which seamlessly connects discrete data and the continuous signal functions in the shared latent space.

Abstract

Recent studies have introduced a new class of generative models for synthesizing implicit neural representations (INRs) that capture arbitrary continuous signals in various domains. These models opened the door for domain-agnostic generative models, but they often fail to achieve high-quality generation. We observed that the existing methods generate the weights of neural networks to parameterize INRs and evaluate the network with fixed positional embeddings (PEs). Arguably, this architecture limits the expressive power of generative models and results in low-quality INR generation. To address this limitation, we propose Domain-agnostic Latent Diffusion Model for INRs (DDMI) that generates adaptive positional embeddings instead of neural networks' weights. Specifically, we develop a Discrete-to-continuous space Variational AutoEncoder (D2C-VAE), which seamlessly connects discrete data and the continuous signal functions in the shared latent space. Additionally, we introduce a novel conditioning mechanism for evaluating INRs with the hierarchically decomposed PEs to further enhance expressive power. Extensive experiments across four modalities, e.g., 2D images, 3D shapes, Neural Radiance Fields, and videos, with seven benchmark datasets, demonstrate the versatility of DDMI and its superior performance compared to the existing INR generative models.

Figures (14)

• Figure 1: Generation results of DDMI. Our DDMI generates high-quality samples across four distinct domains including image, shape, video, and Neural Radiance Fields. DDMI also shows remarkable results for applications like arbitrary-scale image generation or text-to-shape generation.
• Figure 2: Overall pipeline of DDMI. Discrete data $\mathbf{x}$ and continuous function $\boldsymbol{\omega}$ are connected in the shared latent space $\mathbf{z}$ (D2C-VAE). The decoder generates Hierarchically-Decomposed Basis Fields (HDBFs) given latent variable $\mathbf{z}$. $\mathbf{p}^1$ represents the coarsest scale PE and $\mathbf{p}^3$ corresponds to the finest scale PE. The MLP returns the signal value for queried coordinate $c$ using the Coarse-to-Fine Conditioning method. Latent diffusion model operates on the shared latent space. Note that we use a tri-plane latent variable for 3D and video, and a single plane for 2D image.
• Figure 3: Comparison between weight generation and PE generation for INR generative models $G$.$c$ is a coordinate, p is a PE, $\gamma$ is a function that maps coordinates to PEs, $\pi_\theta$ is MLP, and $\hat{\boldsymbol{\omega}}(c)$ is a predicted signal value. For PE generation, we sample basis fields $\Xi$ from $G$ instead of $\theta$. The red line indicates the generation.
• Figure 4: Comparison between DDMI and ScaleParty on arbitrary-scale generation.
• Figure 5: Comparison between DDMI and CIPS on arbitrary-scale generation.