Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kernel KMeans clustering splits for end-to-end unsupervised decision trees

Louis Ohl, Pierre-Alexandre Mattei, Mickaël Leclercq, Arnaud Droit, Frédéric Precioso

TL;DR

Kauri introduces an end-to-end unsupervised binary decision tree that directly optimises a centroid-free kernel KMeans objective, enabling interpretable clustering without relying on external centroid initialisations. By re-expressing the kernel KMeans objective through kernel stocks and greedy splits with cluster-aware gains, it yields competitive clustering performance while often producing shallower, more interpretable trees, especially for non-linear kernels. The method demonstrates robustness across kernels, avoids empty clusters common in kernel KMeans, and provides a practical framework for explainable clustering that integrates seamlessly with kernel methods. This approach advances unsupervised tree learning by delivering an interpretable, end-to-end clustering model with broad kernel compatibility and favorable trade-offs between accuracy and explicability.

Abstract

Trees are convenient models for obtaining explainable predictions on relatively small datasets. Although there are many proposals for the end-to-end construction of such trees in supervised learning, learning a tree end-to-end for clustering without labels remains an open challenge. As most works focus on interpreting with trees the result of another clustering algorithm, we present here a novel end-to-end trained unsupervised binary tree for clustering: Kauri. This method performs a greedy maximisation of the kernel KMeans objective without requiring the definition of centroids. We compare this model on multiple datasets with recent unsupervised trees and show that Kauri performs identically when using a linear kernel. For other kernels, Kauri often outperforms the concatenation of kernel KMeans and a CART decision tree.

Kernel KMeans clustering splits for end-to-end unsupervised decision trees

TL;DR

Kauri introduces an end-to-end unsupervised binary decision tree that directly optimises a centroid-free kernel KMeans objective, enabling interpretable clustering without relying on external centroid initialisations. By re-expressing the kernel KMeans objective through kernel stocks and greedy splits with cluster-aware gains, it yields competitive clustering performance while often producing shallower, more interpretable trees, especially for non-linear kernels. The method demonstrates robustness across kernels, avoids empty clusters common in kernel KMeans, and provides a practical framework for explainable clustering that integrates seamlessly with kernel methods. This approach advances unsupervised tree learning by delivering an interpretable, end-to-end clustering model with broad kernel compatibility and favorable trade-offs between accuracy and explicability.

Abstract

Trees are convenient models for obtaining explainable predictions on relatively small datasets. Although there are many proposals for the end-to-end construction of such trees in supervised learning, learning a tree end-to-end for clustering without labels remains an open challenge. As most works focus on interpreting with trees the result of another clustering algorithm, we present here a novel end-to-end trained unsupervised binary tree for clustering: Kauri. This method performs a greedy maximisation of the kernel KMeans objective without requiring the definition of centroids. We compare this model on multiple datasets with recent unsupervised trees and show that Kauri performs identically when using a linear kernel. For other kernels, Kauri often outperforms the concatenation of kernel KMeans and a CART decision tree.
Paper Structure (33 sections, 51 equations, 7 figures, 11 tables, 3 algorithms)

This paper contains 33 sections, 51 equations, 7 figures, 11 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Training trees
  3. How do we train supervised trees?
  4. How do we train unsupervised trees?
  5. Kauri: KMeans as unsupervised reward ideal
  6. Objective function
  7. Tree branching
  8. Gain metrics
  9. Experiments
  10. General setup
  11. Performances with as many leaves as clusters
  12. Performances with more leaves than clusters
  13. Varying the kernel
  14. A qualitative example of the obtained decision tree
  15. Final words
  16. ...and 18 more sections

Figures (7)

  • Figure 1: A toy example with a dataset consisting of 11 samples partitioned in 3 clusters using 5 leaves in a tree. The matrix represents the kernel between all pairs of samples and dashed areas correspond to the sum of kernel elements according to the kernel stock function $\sigma$.
  • Figure 2: Variations of WAES scores for aligned isotropic 2d Gaussian distributions separated by Kauri or KMeans+Tree as the angle of the alignment (red line in \ref{['sfig:synthetic_dataset_rotation']}) with the x-axis (blue line in \ref{['sfig:synthetic_dataset_rotation']}) $\theta$ grows or the number of samples increases over 30 runs. The distance between the means is $\sqrt{2}$ and the scale matrices are $0.2\pmb{I}_2$.
  • Figure 3: PCA of the wine dataset with samples coloured according to clusters found by Kauri or KMeans+DT with a $\chi^2$ kernel.
  • Figure 4: Number of non-empty clusters for 100 runs of kernel KMeans with an additive $\chi^2$ or polynomial kernel. The algorithm had to find the same number of clusters as classes per dataset.
  • Figure 5: The unsupervised Kauri tree for 2 clusters on the Congressional votes dataset. SA stands for the El Salvador Aid vote, NC for the Nicaraguan Contras vote and MX for the MX-missile vote. The question mark means that the voter did not vote or was missing. Nodes contain their name, the associated cluster to which they assign samples and the type of split that occurred during learning. See Section \ref{['ssec:gain_metrics']} for the split notations and App. \ref{['app:kauri_gains']} for their computation.
  • ...and 2 more figures