Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spectral Analysis of Molecular Kernels: When Richer Features Do Not Guarantee Better Generalization

Asma Jamali, Tin Sum Cheng, Rodrigo A. Vargas-Hernández

TL;DR

This study provides the first comprehensive spectral analysis of molecular kernels applied to QM9 across fingerprint, transformer-based, and 3D representations. It shows that richer spectral content does not consistently translate into better generalization, with many representations exhibiting negative or weak correlations between spectral richness and predictive performance. By employing truncated kernel ridge regression, the authors demonstrate that retaining only the top eigenvalues often recovers the majority of performance, highlighting the predominance of leading spectral components. The findings offer nuanced guidance for evaluating kernel and SSL approaches in data-limited chemical tasks and underscore representation-dependent effects on kernel spectra and generalization.

Abstract

Understanding the spectral properties of kernels offers a principled perspective on generalization and representation quality. While deep models achieve state-of-the-art accuracy in molecular property prediction, kernel methods remain widely used for their robustness in low-data regimes and transparent theoretical grounding. Despite extensive studies of kernel spectra in machine learning, systematic spectral analyses of molecular kernels are scarce. In this work, we provide the first comprehensive spectral analysis of kernel ridge regression on the QM9 dataset, molecular fingerprint, pretrained transformer-based, global and local 3D representations across seven molecular properties. Surprisingly, richer spectral features, measured by four different spectral metrics, do not consistently improve accuracy. Pearson correlation tests further reveal that for transformer-based and local 3D representations, spectral richness can even have a negative correlation with performance. We also implement truncated kernels to probe the relationship between spectrum and predictive performance: in many kernels, retaining only the top 2% of eigenvalues recovers nearly all performance, indicating that the leading eigenvalues capture the most informative features. Our results challenge the common heuristic that "richer spectra yield better generalization" and highlight nuanced relationships between representation, kernel features, and predictive performance. Beyond molecular property prediction, these findings inform how kernel and self-supervised learning methods are evaluated in data-limited scientific and real-world tasks.

Spectral Analysis of Molecular Kernels: When Richer Features Do Not Guarantee Better Generalization

TL;DR

This study provides the first comprehensive spectral analysis of molecular kernels applied to QM9 across fingerprint, transformer-based, and 3D representations. It shows that richer spectral content does not consistently translate into better generalization, with many representations exhibiting negative or weak correlations between spectral richness and predictive performance. By employing truncated kernel ridge regression, the authors demonstrate that retaining only the top eigenvalues often recovers the majority of performance, highlighting the predominance of leading spectral components. The findings offer nuanced guidance for evaluating kernel and SSL approaches in data-limited chemical tasks and underscore representation-dependent effects on kernel spectra and generalization.

Abstract

Understanding the spectral properties of kernels offers a principled perspective on generalization and representation quality. While deep models achieve state-of-the-art accuracy in molecular property prediction, kernel methods remain widely used for their robustness in low-data regimes and transparent theoretical grounding. Despite extensive studies of kernel spectra in machine learning, systematic spectral analyses of molecular kernels are scarce. In this work, we provide the first comprehensive spectral analysis of kernel ridge regression on the QM9 dataset, molecular fingerprint, pretrained transformer-based, global and local 3D representations across seven molecular properties. Surprisingly, richer spectral features, measured by four different spectral metrics, do not consistently improve accuracy. Pearson correlation tests further reveal that for transformer-based and local 3D representations, spectral richness can even have a negative correlation with performance. We also implement truncated kernels to probe the relationship between spectrum and predictive performance: in many kernels, retaining only the top 2% of eigenvalues recovers nearly all performance, indicating that the leading eigenvalues capture the most informative features. Our results challenge the common heuristic that "richer spectra yield better generalization" and highlight nuanced relationships between representation, kernel features, and predictive performance. Beyond molecular property prediction, these findings inform how kernel and self-supervised learning methods are evaluated in data-limited scientific and real-world tasks.
Paper Structure (29 sections, 2 theorems, 25 equations, 12 figures, 3 tables)

This paper contains 29 sections, 2 theorems, 25 equations, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Contribution
  3. Organization
  4. Background
  5. Molecular Representation
  6. Kernel Ridge Regression
  7. Truncated Kernel Ridge Regression
  8. Self-Supervised Learning
  9. Result
  10. Molecular Representations
  11. Discussion
  12. Insights for Kernel Theory and Self-Supervised Learning
  13. Insights for Molecular Chemistry
  14. Transformer-based Representations
  15. 3D Descriptors
  16. ...and 14 more sections

Key Result

Proposition 8

Suppose the integers $m\geq n$ and $p\geq q$. Then for any matrix ${\bm{A}}\in\mathbb{R}^{m\times n}$, we have

Figures (12)

  • Figure 1: Molecular representation generation workflow compatible with kernel functions; (a) global ($\phi({\cal M})$) and (b) local ($\phi_\ell({\cal M})$) molecular representations, where ${\cal M}$ represents a molecule.
  • Figure 2: Test mean absolute error (MAE), computed on $10,000$ molecules, as a function of training set size for three properties: (a) $C_V$, (b) U$_{298}$, and (c) ZPVE. In all panels, results are shown for the Laplacian kernel applied to three global representations (CM, BOB, and SELFIESTED; $\sigma_\ell=10^4$) and one local representation (ACSF; $\sigma_\ell=100$). Solid: Laplacian, and Dashed: Gaussian kernel.
  • Figure 3: Kernel eigenvalue spectra with insets highlighting that nearly half of the eigenvalues are close to zero (main plots) for different molecular representations. Results are shown for (a) ECFPs, (b) SELFIESTED, and (c) local 3D descriptor-based kernels.
  • Figure 4: Correlation between spectral metrics and average R$^2$ across molecular kernel categories. Dotted lines indicate the best-fit linear trend for each category.
  • Figure 5: R$^2$ score for the heat capacity ($C_V$) property as a function of truncation level ($\mu(\%$) for (a) selected ECFP-based kernels and (b) four various global (CM, BOB, SELFIESTED, MLT-BERT) kernel and a single local (ACSF) kernel, all with a Gaussian kernel with $\sigma_\ell= 100$). Left sub-panel: without regularization; right sub-panel: with regularization.
  • ...and 7 more figures

Theorems & Definitions (11)

  • Definition 1: Global Molecular Representation
  • Definition 2: Local Molecular Representation
  • Definition 3: Global Molecular Kernel
  • Definition 4: Local Molecular Kernel
  • Definition 5: Power Law Decay or polynomial decay rate
  • Definition 6: $p$-Stable rank
  • Definition 7: Intrinsic dimension (ID) and stable rank (SR)
  • Proposition 8: Remark 5.4 in ipsen2024stable
  • Definition 9: Spectral Shannon Entropy (SSE), Definition 2.1 in huh2023low
  • Theorem 10
  • ...and 1 more