Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Phase Transitions in Unsupervised Feature Selection

Jonathan Fiorentino, Michele Monti, Dimitrios Miltiadis-Vrachnos, Vittorio Del Tatto, Alessandro Laio, Gian Gaetano Tartaglia

TL;DR

The paper addresses identifying minimal, informative feature sets in high-dimensional protein representations under limited data. It employs Differentiable Information Imbalance ($DII$) as an unsupervised order parameter, analyzing how information content evolves with the retained feature count $F$ across physico-chemical and structure-based descriptors. Key findings show a glass-like transition for physico-chemical features, with a bimodal $P(DII|N,F)$ and a Binder-cumulant minimum, and a corresponding critical point $F_c$ that coincides with the saturation of downstream binary classification performance; structure-based features exhibit a weaker, variance-driven crossover with no sharp $F_c$. This work links feature-space geometry to generalization, revealing feature-type–dependent criticality and providing a principled unsupervised criterion for minimal feature subsets in protein classification, with code and data available at the authors' repository and Zenodo.

Abstract

Identifying minimal and informative feature sets is a central challenge in data analysis, particularly when few data points are available. Here we present a theoretical analysis of an unsupervised feature selection pipeline based on the Differentiable Information Imbalance (DII). We consider the specific case of structural and physico-chemical features describing a set of proteins. We show that if one considers the features as coordinates of a (hypothetical) statistical physics model, this model undergoes a phase transition as a function of the number of retained features. For physico-chemical descriptors, this transition is between a glass-like phase when the features are few and a liquid-like phase. The glass-like phase exhibits bimodal order-parameter distributions and Binder cumulant minima. In contrast, for structural descriptors the transition is less sharp. Remarkably, for physico-chemical descriptors the critical number of features identified from the DII coincides with the saturation of downstream binary classification performance. These results provide a principled, unsupervised criterion for minimal feature sets in protein classification and reveal distinct mechanisms of criticality across different feature types.

Phase Transitions in Unsupervised Feature Selection

TL;DR

The paper addresses identifying minimal, informative feature sets in high-dimensional protein representations under limited data. It employs Differentiable Information Imbalance () as an unsupervised order parameter, analyzing how information content evolves with the retained feature count across physico-chemical and structure-based descriptors. Key findings show a glass-like transition for physico-chemical features, with a bimodal and a Binder-cumulant minimum, and a corresponding critical point that coincides with the saturation of downstream binary classification performance; structure-based features exhibit a weaker, variance-driven crossover with no sharp . This work links feature-space geometry to generalization, revealing feature-type–dependent criticality and providing a principled unsupervised criterion for minimal feature subsets in protein classification, with code and data available at the authors' repository and Zenodo.

Abstract

Identifying minimal and informative feature sets is a central challenge in data analysis, particularly when few data points are available. Here we present a theoretical analysis of an unsupervised feature selection pipeline based on the Differentiable Information Imbalance (DII). We consider the specific case of structural and physico-chemical features describing a set of proteins. We show that if one considers the features as coordinates of a (hypothetical) statistical physics model, this model undergoes a phase transition as a function of the number of retained features. For physico-chemical descriptors, this transition is between a glass-like phase when the features are few and a liquid-like phase. The glass-like phase exhibits bimodal order-parameter distributions and Binder cumulant minima. In contrast, for structural descriptors the transition is less sharp. Remarkably, for physico-chemical descriptors the critical number of features identified from the DII coincides with the saturation of downstream binary classification performance. These results provide a principled, unsupervised criterion for minimal feature sets in protein classification and reveal distinct mechanisms of criticality across different feature types.
Paper Structure (4 sections, 4 figures)

This paper contains 4 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Results
  3. Discussion
  4. Acknowledgments

Figures (4)

  • Figure 1: Criticality in the Differentiable Information Imbalance during feature elimination. (A,B) Average DII versus the number of non-zero features $F$ for the LLPS dataset, for physico-chemical (A) and structural (B) features. (C,D) Heatmaps of the log-transformed probability density of the DII $P(DII|N,F)$ for the LLPS dataset, for $N=54$, for physico-chemical (C) and structural (D) features. DII values are computed on independent test random subsamples using the trained optimal weights.
  • Figure 2: Binder cumulant analysis reveals a glass-like phase transition for physico-chemical features. (A,C) Binder cumulant $U(F)$ as a function of the number of non-zero features $F$ for physico-chemical (A) and structural (C) descriptors, for the LLPS dataset. (B,D) Extrapolation of $F_{min}$ (position of Binder minimum) versus $1/N$, indicating the critical feature number $F_c$ for physico-chemical (B) and structural features (D). (E) DII as a function of the number of non-zero features, computed on the full LLPS dataset ($N=6489$ proteins), for physico-chemical (blue) and structural (orange) features. The blue and orange lines indicate the values of $F_c$ for the two sets of features.
  • Figure 3: Feature-set dependent origin of the phase transition. (A,C) Mean absolute off-diagonal correlation of the physico-chemical (A) and structural (C) feature matrix as a function of the tuning parameter $\beta$. (B,D) Binder cumulant curves $U(F)$ for varying $\beta$ values at fixed $N=162$, for physico-chemical (B) and structural (D) features. The insets show the minimum value of $U(F)$ as a function of $\beta$ at $N=162$. The red line indicates $\beta=0$.
  • Figure 4: Binary classification performance saturates at the critical point for physico-chemical features. (A,B) Area under the receiver operating characteristic (AUROC) as a function of the number of non-zero features $F$ for the LLPS dataset, using physico-chemical (A) and structural (B) features. Solid lines show the mean over subsamples, shadowed areas the standard deviation. Colored dashed vertical lines indicate the values of $F_{min}(N)$ extracted from the Binder cumulant analysis. The black dashed horizontal line denotes the performance of a random classifier. (C,D) Numerical derivative of the AUROC with respect to $F$ for physico-chemical (C) and structural (D) features. AUROC is computed on a held out test set (Supplemental Material Secs. I and VIII). The black dashed horizontal line shows where the derivative is zero.