Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Deep Clustering Algorithms on Non-Categorical 3D CAD Models

Siyuan Xiang, Chin Tseng, Congcong Wen, Deshana Desai, Yifeng Kou, Binil Starly, Daniele Panozzo, Chen Feng

TL;DR

This work tackles the challenge of evaluating deep clustering on non-categorical 3D CAD models by creating a scalable pairwise similarity annotation workflow for a subset of the ABC dataset, yielding 252{,}648 labeled edges across 22{,}968 shapes. It adapts seven baseline clustering methods to 3D data (point clouds and multi-view representations) and introduces an ensemble-based evaluation protocol to cope with sparse ground-truth annotations. The study demonstrates that two-stage baselines (e.g., AtlasNet, BYOL) can outperform some end-to-end deep clustering approaches (e.g., DEC, SCAN, DeepCluster, IIC) in this domain, while highlighting initialization bias and the need for ensemble methods to obtain robust rankings. By releasing the annotation tools, baselines, and evaluation scripts, the work provides a practical benchmark for guiding future methods and enabling scalable analysis of large 3D shape collections in deep geometric computing.

Abstract

We introduce the first work on benchmarking and evaluating deep clustering algorithms on large-scale non-categorical 3D CAD models. We first propose a workflow to allow expert mechanical engineers to efficiently annotate 252,648 carefully sampled pairwise CAD model similarities, from a subset of the ABC dataset with 22,968 shapes. Using seven baseline deep clustering methods, we then investigate the fundamental challenges of evaluating clustering methods for non-categorical data. Based on these challenges, we propose a novel and viable ensemble-based clustering comparison approach. This work is the first to directly target the underexplored area of deep clustering algorithms for 3D shapes, and we believe it will be an important building block to analyze and utilize the massive 3D shape collections that are starting to appear in deep geometric computing.

Evaluating Deep Clustering Algorithms on Non-Categorical 3D CAD Models

TL;DR

This work tackles the challenge of evaluating deep clustering on non-categorical 3D CAD models by creating a scalable pairwise similarity annotation workflow for a subset of the ABC dataset, yielding 252{,}648 labeled edges across 22{,}968 shapes. It adapts seven baseline clustering methods to 3D data (point clouds and multi-view representations) and introduces an ensemble-based evaluation protocol to cope with sparse ground-truth annotations. The study demonstrates that two-stage baselines (e.g., AtlasNet, BYOL) can outperform some end-to-end deep clustering approaches (e.g., DEC, SCAN, DeepCluster, IIC) in this domain, while highlighting initialization bias and the need for ensemble methods to obtain robust rankings. By releasing the annotation tools, baselines, and evaluation scripts, the work provides a practical benchmark for guiding future methods and enabling scalable analysis of large 3D shape collections in deep geometric computing.

Abstract

We introduce the first work on benchmarking and evaluating deep clustering algorithms on large-scale non-categorical 3D CAD models. We first propose a workflow to allow expert mechanical engineers to efficiently annotate 252,648 carefully sampled pairwise CAD model similarities, from a subset of the ABC dataset with 22,968 shapes. Using seven baseline deep clustering methods, we then investigate the fundamental challenges of evaluating clustering methods for non-categorical data. Based on these challenges, we propose a novel and viable ensemble-based clustering comparison approach. This work is the first to directly target the underexplored area of deep clustering algorithms for 3D shapes, and we believe it will be an important building block to analyze and utilize the massive 3D shape collections that are starting to appear in deep geometric computing.
Paper Structure (22 sections, 11 figures)

This paper contains 22 sections, 11 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Annotation Creation
  4. Human annotation
  5. Annotation creation workflow
  6. Design Decisions on the Annotation Workflow
  7. Benchmark
  8. Baseline methods
  9. Evaluation Metrics
  10. Ensemble-based Evaluation
  11. Experiments and Discussions
  12. Limitations and discussion
  13. Conclusion
  14. Appendix
  15. Challenges for class annotation
  16. ...and 7 more sections

Figures (11)

  • Figure 1: The overview of our work via a subset of clustering results from a baseline (DeepCluster caron2018deep). It demonstrates the challenges of classification-based labeling in our task, due to many non-standard mechanical components (green). Each section shows some random CAD models in the same cluster. A red section shows a cluster with annotation violations highlighted at the red objects.
  • Figure 2: Human annotation workflow.
  • Figure 3: Evaluation bias when using Human alone, based on different cluster initialization from MVCNN (option-A) and AtlasNet (option-B).
  • Figure 4: Consistent ensemble-based external evaluation indices. Rank of balanced accuracy for both Ensemble and EnsembleHuman: AtlasNet $>$ SCAN $>$ DEC $>$ IIC $>$ BYOL $>$ DeepCluster $>$ MVCNN.
  • Figure 5: Benchmark results of internal evaluation indices. Chamfer ranking: AtlasNet $>$ SCAN $>$ DEC $>$ BYOL $>$ DeepCluster $>$ IIC $>$ MVCNN; Jaccard ranking: AtlasNet $>$ DEC $>$ SCAN $>$ BYOL $>$ IIC $>$ DeepCluster $>$ MVCNN.
  • ...and 6 more figures