Shape Generation via Weight Space Learning

TL;DR

This paper addresses the challenge of leveraging rich geometric priors in 3D shape generation when data are scarce or noisy by treating the weight space $\\mathbb{R}^W$ of a large shape-generative model as a data modality. It proposes weight-space learning, where conditioning induces a manifold in weight space and small, structured movements along this manifold can separately modulate global topology and local geometry without full fine-tuning. Through two experiments, it demonstrates a global connectivity phase transition triggered by weight-space interpolation and shows that low-dimensional, PCA-based subspaces of conditioning vectors can yield diverse local refinements while preserving topology. The work suggests a path toward robust, data-efficient, geometry-aware generation and sets the stage for broader comparisons across architectures and deeper mechanistic understanding of weight-space sensitivities.

Abstract

Foundation models for 3D shape generation have recently shown a remarkable capacity to encode rich geometric priors across both global and local dimensions. However, leveraging these priors for downstream tasks can be challenging as real-world data are often scarce or noisy, and traditional fine-tuning can lead to catastrophic forgetting. In this work, we treat the weight space of a large 3D shape-generative model as a data modality that can be explored directly. We hypothesize that submanifolds within this high-dimensional weight space can modulate topological properties or fine-grained part features separately, demonstrating early-stage evidence via two experiments. First, we observe a sharp phase transition in global connectivity when interpolating in conditioning space, suggesting that small changes in weight space can drastically alter topology. Second, we show that low-dimensional reparameterizations yield controlled local geometry changes even with very limited data. These results highlight the potential of weight space learning to unlock new approaches for 3D shape generation and specialized fine-tuning.

Figures (4)

• Figure 1: Conceptual overview. 3D shape-generative foundation models encode geometric priors (both global and local) in their weights. We hypothesize that these can be leveraged through weight space learning in a controlled fashion to address downstream goals.
• Figure 2: Hypothesis (schematic). Structured modulations of the weight space $\mathbb{R}^W$ can isolate global vs. local shape changes and can be learned, enabling targeted geometry manipulations.
• Figure 3: Phase transition in connectivity. By interpolating between evenly and unevenly sampled point clouds on the unit sphere (top row) from $\alpha=0$ to $\alpha=1$, we observe a sudden jump in the generated shapes (middle row) from a single connected component to many disconnected parts, indicating that minimal changes in weight space can induce abrupt global topological shifts (bottom plot).
• Figure 4: Generative modeling of scarce data (SimJEB). We sample point clouds from $k=381$ unseen SimJEB bracket shapes to obtain conditions $\mathbf{c}_i$ (left). We extract principal components, and sample new weights $\boldsymbol{\theta}_{\tilde{\mathbf{c}}}$ within this PCA subspace (center). Ultimately, we can produce novel bracket designs that preserve overall topology yet exhibit distinct local geometry (right).