HAL3D: Hierarchical Active Learning for Fine-Grained 3D Part Labeling

TL;DR

HAL3D introduces a hierarchical, symmetry-aware active learning framework for fine-grained 3D part labeling that tightly couples a deep label proposal network with human verification and modification in a self-improving loop. By organizing labels into a tree of AND/OR nodes and exploiting symmetry, HAL3D achieves near-full labeling accuracy while dramatically reducing human effort, outperforming fully automatic baselines and enabling efficient annotation on PartNet and ABO datasets. Key contributions include the label proposal network design, the hierarchical labeling strategy, and symmetry-assisted acceleration, with empirical results showing substantial labeling-time reductions and strong accuracy. The work advances practical 3D understanding by enabling scalable, interactive, fine-grained part labeling that can support applications in quality control, assembly, and robotics.

Abstract

We present the first active learning tool for fine-grained 3D part labeling, a problem which challenges even the most advanced deep learning (DL) methods due to the significant structural variations among the small and intricate parts. For the same reason, the necessary data annotation effort is tremendous, motivating approaches to minimize human involvement. Our labeling tool iteratively verifies or modifies part labels predicted by a deep neural network, with human feedback continually improving the network prediction. To effectively reduce human efforts, we develop two novel features in our tool, hierarchical and symmetry-aware active labeling. Our human-in-the-loop approach, coined HAL3D, achieves 100% accuracy (barring human errors) on any test set with pre-defined hierarchical part labels, with 80% time-saving over manual effort.

Figures (7)

• Figure 1: Fine-grained parts of the 3D models shown exhibit the kind of geometric and structural complexity and diversity that part labeling has to handle. No existing methods, whether learned or heuristic-based, could obtain close-to-fully-accurate labeling in these challenging cases, while our human-in-the-loop active learning tool can approach full accuracy, barring human errors.
• Figure 2: Our human-in-the-loop, hierarchical active learning (HAL3D) tool for fine-grained 3D part labeling. The input consists of a set of test shapes each pre-segmented into parts. The labeling proceeds hierarchically, following a tree structure (left) that organizes the hierarchical part labels, from coarse labels (top) to fine-grained labels (bottom). For a node in the label tree (with label $l$ = "Regular Leg Base" in the illustration), the input $I_0$ is the subset of parts, from the entire set of shapes, that are labeled $l$ by its parent. When labeling parts with $l$, the label proposal module first assigns refined labels for each part. Then, proposals are sorted by the mean label probability over the parts of each shape, with the high-confidence (HC) proposals passed to the verification step, which stops once the rate of failed shapes reaches a threshold. The low-confidence (LC) proposals are passed to the label modification module. Correctly labeled shapes fine-tune the network, while skipped and failed shapes, in set $I_1$, go to the next iteration for labeling. The iterations terminate when all shapes have passed human validation.
• Figure 3: Label proposal architectures. Part label prediction is based on global and local, part-level features at the "AND" nodes and global features only for the "OR" nodes. The "AND" nodes would serve to predict the label for each part and the "OR" nodes would predict labels for the entire input shape. Notion-wise, $N$ is the number of points, $P$ is the number of parts, $K$ is the feature dimension, and $L$ is the number of labels.
• Figure 4: Comparison of pre-segmented parts in PartNet and ABO. ABO shapes were pre-segmented into connected components where each component may correspond to multiple labels, e.g., the back and the arm of the chair are grouped into the same component, as indicated by the red color in the second column. Instead, our part labeling operates on the convex pieces obtained by a decomposition. In the first three figures, the colors are used to distinguish between different parts, and we can see that the parts in PartNet are manually segmented and exhibiting imperfect boundaries. In the last figure, the colors reflect semantic labels.
• Figure 5: Visual comparison between labeling methods. HAL3D can achieve results approaching GT, barring human errors. The only mis-labeled part by HAL3D here is the bottom panel of the table (red oval), which could be easily labeled as shelf by users.