Memorization in 3D Shape Generation: An Empirical Study

TL;DR

This work tackles memorization in 3D shape generation by designing an evaluation framework that separates true generalization from data copying. It defines object-level replicas and model-level memorization using the Mann-Whitney $Z_U$ statistic alongside Fréchet Distance $FD$ to control for output quality, applying this to a range of existing models. Through controlled Vecset-diffusion experiments, it reveals that memorization is more pronounced for image data than for 3D data, increases with data diversity and conditioning granularity, and peaks at moderate guidance scales. The study further shows that longer latent Vecsets and rotation augmentation can mitigate memorization without sacrificing generation fidelity, offering practical guidelines for building more generalizable 3D generators.

Abstract

Generative models are increasingly used in 3D vision to synthesize novel shapes, yet it remains unclear whether their generation relies on memorizing training shapes. Understanding their memorization could help prevent training data leakage and improve the diversity of generated results. In this paper, we design an evaluation framework to quantify memorization in 3D generative models and study the influence of different data and modeling designs on memorization. We first apply our framework to quantify memorization in existing methods. Next, through controlled experiments with a latent vector-set (Vecset) diffusion model, we find that, on the data side, memorization depends on data modality, and increases with data diversity and finer-grained conditioning; on the modeling side, it peaks at a moderate guidance scale and can be mitigated by longer Vecsets and simple rotation augmentation. Together, our framework and analysis provide an empirical understanding of memorization in 3D generative models and suggest simple yet effective strategies to reduce it without degrading generation quality. Our code is available at https://github.com/zlab-princeton/3d_mem.

Figures (26)

• Figure 1: Examples of generated 3D shapes that illustrate memorization vs. generalization relative to the training shapes. In this paper, we propose a framework to evaluate memorization in 3D shape generation, use it to quantify memorization in existing methods, and conduct controlled experiments to study how data and modeling designs impact memorization.
• Figure 2: Generated samples from the baseline model at each decile, ranked by LFD to the nearest training shape.
• Figure 3: Training dynamics of the baseline model. As training progresses, $Z_U$ and training FD simultaneously decrease, whereas test FD decreases initially before plateauing at around 200K steps.
• Figure 4: Fine-grained conditioning increases memorization. For both class-conditional (left) and text-conditional (right) models, as the conditioning becomes finer-grained, the 3D generative models exhibit stronger memorization.
• Figure 5: Higher data diversity increases memorization. With a fixed training set size, increasing the number of classes from 16 to 100 leads to a moderate increase in memorization.