Sparse clustering via the Deterministic Information Bottleneck algorithm

TL;DR

Addresses clustering when informative signal is confined to a small subset of features in high-dimensional data. The authors propose Sparse DIB, an extension of the deterministic Information Bottleneck that learns a clustering while simultaneously weighting features via a nonnegative vector $\mathbf{w}$ under $L_1$ and $L_2$ constraints, using a weighted perturbed similarity matrix $P^{'W}_{Y|X}$ and alternating optimization with Dykstra projection. The approach is validated through a simulation study across varying dimensionality and sparsity, benchmarked against six sparse clustering methods, and demonstrated on a bladder cancer transcriptomics dataset where Sparse DIB yields interpretable feature subsets and competitive clustering performance. The work advances information-theoretic clustering for sparse, high-dimensional data with interpretable feature selection, with potential extensions to hierarchical, cluster-specific weighting, and mixed-type data.

Abstract

Cluster analysis relates to the task of assigning objects into groups which ideally present some desirable characteristics. When a cluster structure is confined to a subset of the feature space, traditional clustering techniques face unprecedented challenges. We present an information-theoretic framework that overcomes the problems associated with sparse data, allowing for joint feature weighting and clustering. Our proposal constitutes a competitive alternative to existing clustering algorithms for sparse data, as demonstrated through simulations on synthetic data. The effectiveness of our method is established by an application on a real-world genomics data set.

Figures (3)

• Figure 1: Box plots of ARI values across different dimensionalities ($p$) and ratios of relevant features ($q$) for each method.
• Figure 2: Trajectories of normalised entropy for sparsity parameter $u \in [0.4, 10]$. The black points mark the value of $u$ and the corresponding normalised entropy where the number of non-zero weights is closest to $\rho = \lfloor p \times q \rfloor$.
• Figure 3: Genes with non-zero weights selected by Sparse DIB marked by type.