TFDM: Time-Variant Frequency-Based Point Cloud Diffusion with Mamba

TL;DR

This work tackles the computational bottlenecks of diffusion-based 3D point cloud generation by introducing TFDM, a framework that combines time-variant frequency-based encoding with latent space diffusion powered by Mamba state-space models. The core innovations are dual latent Mamba blocks arranged with space-filling curve serialization and a TF-Encoder that emphasizes high-frequency geometric details at later diffusion steps, enabling detailed yet efficient point cloud generation. Empirical evaluation on ShapeNet-v2 demonstrates state-of-the-art or competitive performance on several metrics, along with substantial reductions in parameters and inference time compared with recent baselines. The approach offers a practical advancement for high-fidelity, scalable 3D generation and suggests future work to integrate frequency analysis more deeply into end-to-end training.

Abstract

Diffusion models currently demonstrate impressive performance over various generative tasks. Recent work on image diffusion highlights the strong capabilities of Mamba (state space models) due to its efficient handling of long-range dependencies and sequential data modeling. Unfortunately, joint consideration of state space models with 3D point cloud generation remains limited. To harness the powerful capabilities of the Mamba model for 3D point cloud generation, we propose a novel diffusion framework containing dual latent Mamba block (DM-Block) and a time-variant frequency encoder (TF-Encoder). The DM-Block apply a space-filling curve to reorder points into sequences suitable for Mamba state-space modeling, while operating in a latent space to mitigate the computational overhead that arises from direct 3D data processing. Meanwhile, the TF-Encoder takes advantage of the ability of the diffusion model to refine fine details in later recovery stages by prioritizing key points within the U-Net architecture. This frequency-based mechanism ensures enhanced detail quality in the final stages of generation. Experimental results on the ShapeNet-v2 dataset demonstrate that our method achieves state-of-the-art performance (ShapeNet-v2: 0.14\% on 1-NNA-Abs50 EMD and 57.90\% on COV EMD) on certain metrics for specific categories while reducing computational parameters and inference time by up to 10$\times$ and 9$\times$, respectively. Source code is available in Supplementary Materials and will be released upon accpetance.

Figures (9)

• Figure 1: 1-NNA-Abs50 EMD & COV EMD ( \ref{['sec:exp-1']}) performance (%) vs. parameter size (millions) on ShapeNet-v2 Car category. For 1-NNA-Abs50 EMD (left), lower value indicates better generation quality and fidelity. For COV EMD (right), higher is better diversity. In both plots, moving left along the horizontal axis denotes smaller model.
• Figure 2: The overview architecture of our proposed TFDM. The network takes a point cloud at timestep $t$ as input and aims to predict the noise component in $\mathcal{X}_t$ to obtain the point cloud at timestep $t-1$. Initially, the input point cloud is passed through a time-variant frequency-based encoder. This is followed by a latent embedding module that generates a latent point cloud $\hat{\mathcal{X}_t}$. The latent point cloud is then processed through Two-Streams Mamba blocks, which apply different serialization methods to extract diverse and complementary features. Subsequently, an affine transformation block is employed to align the latent point clouds from the different streams, ensuring consistency and integration of the extracted features. Finally, the aligned latent representation is decoded back into the 3D space.
• Figure 3: Qualitative results comparing our approach (right) with other leading contemporary approaches (left/middle), our TFDM can generate high-quality and diverse point clouds. Three illustrative object categories $\{airplanes, chairs, cars\}$ are included here only.
• Figure 4: Illustration of the frequency key point selection process within the encoder to show how different strategies are applied across various timelines to obtain a downsampled point cloud. Subsequently, the downsampled point cloud is used to query the latent volume, resulting in the latent point cloud.
• Figure 5: Illustration of our proposed Latent mamba block, which includes Layer Norm, Linear Layer, forward and backward state space model with its corresponding Conv1D block (N.B. we only perform serialization at the first block).