Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLiMB: A Domain-Informed Novelty Detection Clustering Framework for Scientific Discovery

Lorenzo Monti, Tatiana Muraveva, Brian Sheridan, Davide Massari, Alessia Garofalo, Gisella Clementini, Umberto Michelucci

TL;DR

CLiMB introduces a domain-informed, two-phase novelty-detection clustering framework that decouples exploitation of prior knowledge from exploratory discovery. Phase 1 (K-Bound) anchors seed-guided clusters using Density, Distance, and Radial constraints, while Phase 2 (Exploratory) clusters the residual data with density-based methods to reveal arbitrary topologies. Applied to Gaia DR3 RR Lyrae stars, CLiMB achieves a notable ARI of 0.829 for known substructures, vastly outperforming constraint-based and heuristic baselines, and identifies three novel dynamical groups in the unlabelled field, including the Galactic Disk and the Shiva/Shakti structures. The approach demonstrates robust, monotonic gains with increasing supervision and offers a principled framework for multi-phase discovery in high-dimensional scientific datasets, albeit with higher hyperparameter complexity and scalability considerations.

Abstract

In data-driven scientific discovery, a challenge lies in classifying well-characterized phenomena while identifying novel anomalies. Current semi-supervised clustering algorithms do not always fully address this duality, often assuming that supervisory signals are globally representative. Consequently, methods often enforce rigid constraints that suppress unanticipated patterns or require a pre-specified number of clusters, rendering them ineffective for genuine novelty detection. To bridge this gap, we introduce CLiMB (CLustering in Multiphase Boundaries), a domain-informed framework decoupling the exploitation of prior knowledge from the exploration of unknown structures. Using a sequential two-phase approach, CLiMB first anchors known clusters using constrained partitioning, and subsequently applies density-based clustering to residual data to reveal arbitrary topologies. We demonstrate this framework on RR Lyrae stars data from the Gaia Data Release 3. CLiMB attains an Adjusted Rand Index of 0.829 with 90% seed coverage in recovering known Milky Way substructures, drastically outperforming heuristic and constraint-based baselines, which stagnate below 0.20. Furthermore, sensitivity analysis confirms CLiMB's superior data efficiency, showing monotonic improvement as knowledge increases. Finally, the framework successfully isolates three dynamical features (Shiva, Shakti, and the Galactic Disk) in the unlabelled field, validating its potential for scientific discovery.

CLiMB: A Domain-Informed Novelty Detection Clustering Framework for Scientific Discovery

TL;DR

CLiMB introduces a domain-informed, two-phase novelty-detection clustering framework that decouples exploitation of prior knowledge from exploratory discovery. Phase 1 (K-Bound) anchors seed-guided clusters using Density, Distance, and Radial constraints, while Phase 2 (Exploratory) clusters the residual data with density-based methods to reveal arbitrary topologies. Applied to Gaia DR3 RR Lyrae stars, CLiMB achieves a notable ARI of 0.829 for known substructures, vastly outperforming constraint-based and heuristic baselines, and identifies three novel dynamical groups in the unlabelled field, including the Galactic Disk and the Shiva/Shakti structures. The approach demonstrates robust, monotonic gains with increasing supervision and offers a principled framework for multi-phase discovery in high-dimensional scientific datasets, albeit with higher hyperparameter complexity and scalability considerations.

Abstract

In data-driven scientific discovery, a challenge lies in classifying well-characterized phenomena while identifying novel anomalies. Current semi-supervised clustering algorithms do not always fully address this duality, often assuming that supervisory signals are globally representative. Consequently, methods often enforce rigid constraints that suppress unanticipated patterns or require a pre-specified number of clusters, rendering them ineffective for genuine novelty detection. To bridge this gap, we introduce CLiMB (CLustering in Multiphase Boundaries), a domain-informed framework decoupling the exploitation of prior knowledge from the exploration of unknown structures. Using a sequential two-phase approach, CLiMB first anchors known clusters using constrained partitioning, and subsequently applies density-based clustering to residual data to reveal arbitrary topologies. We demonstrate this framework on RR Lyrae stars data from the Gaia Data Release 3. CLiMB attains an Adjusted Rand Index of 0.829 with 90% seed coverage in recovering known Milky Way substructures, drastically outperforming heuristic and constraint-based baselines, which stagnate below 0.20. Furthermore, sensitivity analysis confirms CLiMB's superior data efficiency, showing monotonic improvement as knowledge increases. Finally, the framework successfully isolates three dynamical features (Shiva, Shakti, and the Galactic Disk) in the unlabelled field, validating its potential for scientific discovery.
Paper Structure (30 sections, 2 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 30 sections, 2 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methods
  3. CLiMB: A Domain-Informed Novelty Detection Clustering Algorithm
  4. Phase 1: K-Bound, Constrained Clustering
  5. Advanced Initialization and Metric Flexibility
  6. Initialization Strategies:
  7. Metric Flexibility:
  8. Phase 2: Exploratory Novelty Detection
  9. Convergence, Complexity, and Implementation
  10. Convergence Criteria
  11. Computational Complexity
  12. Implementation Details
  13. Related Works
  14. Evolution of Methodological Paradigms in SSC
  15. Persistent Challenges and Identified Research Gaps
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Distribution of 4,933 RRLs from Gaia DR3 in the $E$-$L_z$ and $L_z$-$L_\perp$ planes, colour-coded by the substructure to which they belong. Grey dots indicate RRLs not assigned to any substructure. The top panels show RRLs identified by the CLiMB algorithm in known MW substructures from Dodd2023, while the bottom panels show substructures not reported by Dodd2023, discovered during the exploratory phase of the CLiMB algorithm.
  • Figure 2: Visual benchmark of clustering results on the known subset of structures. In all panels, distinct colors represent separate cluster assignments. Top-Left: Partial ground truth for the 8 significant substructures. Top-Right: CLiMB output (ARI: 0.829) showing accurate recovery of complex, non-convex shapes. Bottom-Left: Heuristic SS-DBSCAN (ARI: 0.040) fails to form coherent structures, fragmenting streams into noise. Bottom-Right: C-DBSCAN (ARI: 0.152) suffers from massive over-merging, collapsing distinct kinematic streams into a single macro-cluster.
  • Figure 3: Diagnostic plot of CLiMB's final result in the $E$-$L_z$ plane. Points are categorized by both shape and color to distinguish the algorithmic phases: circles in cool colors represent structures recovered during the constrained phase (Phase 1), while crosses in warm colors denote clusters identified during the exploratory phase (Phase 2). This diagnostic validates the detection of the Shiva (orange) and Shakti (yellow) structures, as well as the Galactic Disk (red), all recovered as exploratory discoveries.
  • Figure 4: Sensitivity analysis showing the Adjusted Rand Index (ARI) on knowledge recovery as a function of the percentage of prior knowledge (10% to 100%). CLiMB (blue) demonstrates monotonic learning and high performance even with limited supervision, whereas baselines (C-DBSCAN and SSDBSCAN in orange and red) stagnate.