A Tutorial on Discriminative Clustering and Mutual Information

TL;DR

This tutorial surveys discriminative clustering, contrasting it with generative approaches and tracing the historical role of mutual information as an objective. It shows how learning shifted from fixed decision boundaries toward representation-based and contrastive strategies, culminating in energy-based, InfoMax-inspired formulations that use augmentations and MI lower bounds. It also discusses MI’s limitations, the importance of regularisation, and the need for model-aligned internal metrics and integrated model selection. The GemClus package is presented as a practical demonstration of these ideas, spanning linear, kernelized, and neural discriminative clustering methods. Together, the work highlights a trajectory from explicit probabilistic modelling to flexible, invariant-driven learning paradigms with practical tooling for researchers and developers.

Abstract

To cluster data is to separate samples into distinctive groups that should ideally have some cohesive properties. Today, numerous clustering algorithms exist, and their differences lie essentially in what can be perceived as ``cohesive properties''. Therefore, hypotheses on the nature of clusters must be set: they can be either generative or discriminative. As the last decade witnessed the impressive growth of deep clustering methods that involve neural networks to handle high-dimensional data often in a discriminative manner; we concentrate mainly on the discriminative hypotheses. In this paper, our aim is to provide an accessible historical perspective on the evolution of discriminative clustering methods and notably how the nature of assumptions of the discriminative models changed over time: from decision boundaries to invariance critics. We notably highlight how mutual information has been a historical cornerstone of the progress of (deep) discriminative clustering methods. We also show some known limitations of mutual information and how discriminative clustering methods tried to circumvent those. We then discuss the challenges that discriminative clustering faces with respect to the selection of the number of clusters. Finally, we showcase these techniques using the dedicated Python package, GemClus, that we have developed for discriminative clustering.

Figures (7)

• Figure 1: The generative and discriminative modelling frameworks for clustering models. Observed variables are shaded.
• Figure 2: An example of generative models with 2 clusters in 1 dimension. The parameters $\theta$ of the model comprise the Gaussian distribution locations $\pmb{\mu}_\text{red}=-1$, $\pmb{\mu}_\text{blue}=2$, the scales $\sigma_\text{red}^2=\sigma^2_\text{blue}=1$ and the proportions of each clusters.
• Figure 3: Clustering of mixture of 3 isotropic Gaussian distributions by a nonparametric model. When the cluster memberships are directly optimised by mutual information, the clusters do not correspond to any of the initial Gaussian distributions because the cluster memberships are not derived from the position of the samples. In constrast, MI regularised by maximum mean discrepancy can optimise correctly the nonparametric model. This example is taken from ohl_generalised_2022 and can be reproduced using their package GemClus ohl_gemclus_2023.
• Figure 4: Graphical explanation of notations used for deep clustering models.
• Figure 5: Final clustering and decision boundaries of the example models on a circle dataset. Blue stands for low probability of cluster 2, red for high.

Theorems & Definitions (3)

• Theorem 1: Independence in mutual information
• Remark 1
• Remark 2