Aligning Instance-Semantic Sparse Representation towards Unsupervised Object Segmentation and Shape Abstraction with Repeatable Primitives

TL;DR

The paper tackles the challenge of unsupervised semantic shape representation for 3D objects by proposing a one-stage framework that jointly learns instance segmentation, semantic segmentation, and shape abstraction with repeatable primitives. It leverages Sparse Latent Membership Pursuit (SLMP) to produce sparse, semantically meaningful part features and an Alignment Induced Decoder to align instance- and semantic-level geometry, enabling reconstruction with repeatable deformable superquadrics. Key contributions include integrating Sparsemax-based sparsity, a custom attention-based alignment with adaptive temperature, and cascade unfrozen learning to stabilize semantic-geometry identification, all without category-specific supervision. Experiments on ShapeNet and Animal categories demonstrate competitive or superior performance across semantic/instance segmentation and shape abstraction, while maintaining generalization to new categories and the discovery of repeatable primitives with cross-category training benefits.

Abstract

Understanding 3D object shapes necessitates shape representation by object parts abstracted from results of instance and semantic segmentation. Promising shape representations enable computers to interpret a shape with meaningful parts and identify their repeatability. However, supervised shape representations depend on costly annotation efforts, while current unsupervised methods work under strong semantic priors and involve multi-stage training, thereby limiting their generalization and deployment in shape reasoning and understanding. Driven by the tendency of high-dimensional semantically similar features to lie in or near low-dimensional subspaces, we introduce a one-stage, fully unsupervised framework towards semantic-aware shape representation. This framework produces joint instance segmentation, semantic segmentation, and shape abstraction through sparse representation and feature alignment of object parts in a high-dimensional space. For sparse representation, we devise a sparse latent membership pursuit method that models each object part feature as a sparse convex combination of point features at either the semantic or instance level, promoting part features in the same subspace to exhibit similar semantics. For feature alignment, we customize an attention-based strategy in the feature space to align instance- and semantic-level object part features and reconstruct the input shape using both of them, ensuring geometric reusability and semantic consistency of object parts. To firm up semantic disambiguation, we construct cascade unfrozen learning on geometric parameters of object parts.

Figures (15)

• Figure 1: Unsupervised semantic shape representation via feature alignment in low-dimensional subspaces of a high-dimensional space (sparse representation): instance-level / semantic-level segmentation and shape abstraction w/o or w/ repeatable primitive-reconstructed parts. Each circle represents the geometric feature of an object part.
• Figure 2: Alignment of instance-level geometric features with semantic-level geometric features. Instance-level latent membership pursuit determines the pose and geometry of each individual part, while semantic-level latent membership pursuit captures shared geometry among repeatable parts. Instance-level geometric features introduce variations among the four legs (middle), while semantic-level geometric features ensure consistent and repeatable geometry (right). Note that shape abstraction retains a few defining geometric characteristics, rather than the pursuit of exhaustive shape reconstruction.
• Figure 3: Cascade unfreezing learning on DSQ parameters.
• Figure 4: Overview of our pipeline. $\bigotimes$ represents matrix multiplication. SLMP outputs segmentation and the sparse weight matrix to construct part features in low-dimensional subspaces, refer to Fig. \ref{['fig: sparseLMP']} and Section \ref{['sec: sparse latent membership pursuit']}. Trans bias, Pos-Decoder, and Geo-Decoder are used to predict primitive parameters, refer to Fig. \ref{['fig: decoder']} for details. Sim Func indicates the inner product similarity with an adaptive temperature parameter, defined by (\ref{['eq: attention alignment with temperature']}) and (\ref{['eq: tau']}). Part-wise features include: Instance-level pose features (Pos), Instance-level geometric features (Geo-i), Semantic-level geometric features (Geo-s), and semantic-aligned instance-level geometric features (Geo-is). Outputs, from top to bottom, are: instance segmentation (cf. Instance-level SLMP), instance shape abstraction (cf. Geo-i), semantic shape abstraction (cf. Geo-is), semantic shape abstraction with repeatable parts (cf. Geo-s), and semantic segmentation (cf. Semantic-level SLMP).
• Figure 5: Sparse Latent Membership Pursuit (SLMP). The matrices in the figure have been transposed for presentational convenience.