StructRe: Rewriting for Structured Shape Modeling

TL;DR

StructRe casts structured shape modeling as a local, probabilistic rewriting problem: a shape is encoded into a unified part-space and geometry patches, then rewritten upward to coarser components or downward to finer parts through a learned rewriting network. The framework comprises a patch-based geometry encoder, a unified part-space encoder, and a transformer-based rewriting model that performs iterative decoding with probabilistic sampling, aided by data augmentation to generalize across categories. It demonstrates robust reconstruction and generation across PartNet and ShapeNet, including zero-/few-shot adaptation to ShapeNet, and enables applications such as editing and shape vectorization by parts. The findings show that local rewriting with probabilistic sampling improves generalization, reduces reliance on category priors, and yields regular, detailed, and diverse hierarchies suitable for cross-category shape understanding and manipulation.

Abstract

Man-made 3D shapes are naturally organized in parts and hierarchies; such structures provide important constraints for shape reconstruction and generation. Modeling shape structures is difficult, because there can be multiple hierarchies for a given shape, causing ambiguity, and across different categories the shape structures are correlated with semantics, limiting generalization. We present StructRe, a structure rewriting system, as a novel approach to structured shape modeling. Given a 3D object represented by points and components, StructRe can rewrite it upward into more concise structures, or downward into more detailed structures; by iterating the rewriting process, hierarchies are obtained. Such a localized rewriting process enables probabilistic modeling of ambiguous structures and robust generalization across object categories. We train StructRe on PartNet data and show its generalization to cross-category and multiple object hierarchies, and test its extension to ShapeNet. We also demonstrate the benefits of probabilistic and generalizable structure modeling for shape reconstruction, generation and editing tasks.

Figures (22)

• Figure 1: Shape structure ambiguity. A single shape can have multiple possible decompositions. As highlighted by the red rectangles, the chair back can be decomposed into 2 parts or 5 parts at the same level (level 0), with similar variations observed at level 1 and 2 highlighted by the yellow rectangles, illustrating the structural ambiguity inherent in shape decomposition.
• Figure 2: Similarity of local composition across shapes. From top to bottom: three objects have their bases detected and grouped into single parts (in brown) by a common query token of our rewriting network (c.f. Sec. \ref{['subsec:rewrite_network']}). The rewriting system learns such local compositions for resolving structure ambiguity and achieving robust generalization across object categories.
• Figure 3: Pipeline overview.StructRe consists of three modules that are trained sequentially. (a) first, we learn a compact patch encoding $\vb{p}_i\in\vb{P}$ of detailed geometry by training an autoencoder $(E_p,D_p)$. (b) second, we learn a part shape encoding $\vb{s}\in\vb{\Sigma}$ by training an autoencoder $(E_s,D_s)$ on all parts and shapes, which crucially provides a unified space over which to reason rewriting. (c) third, we learn the rewriting model by training $(E_R,D_R)$ that transits from the input parts $[s_i]$ and patches $[\vb{p}_i]$ to the output parts $[s'_i]$, controlled by the direction $d\in \{\uparrow,\downarrow\}$. The decoder $D_R$ features iterative decoding that refines the output from a previous iteration subsequently. In the illustration, four chair legs are rewrote into a base.
• Figure 4: Masked input for detailed reconstruction. With increasing ratios of $[\vb{p}'_i]$ visible to $D_s$, the recovered shapes start with rough overall geometry (0%) and gradually obtain more fine details (25%, 50%).
• Figure 5: Examples of ablation results. Top row is rewriting in upward direction, bottom row in downward direction. Compared to our full model, the variations on data augmentation, iterative decoding and input geometry conditioning produce less accurate results with missing, duplicated or distorted parts.