Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SHAPCA: Consistent and Interpretable Explanations for Machine Learning Models on Spectroscopy Data

Mingxing Zhang, Nicola Rossberg, Simone Innocente, Katarzyna Komolibus, Rekha Gautam, Barry O'Sullivan, Luca Longo, Andrea Visentin

Abstract

In recent years, machine learning models have been increasingly applied to spectroscopic datasets for chemical and biomedical analysis. For their successful adoption, particularly in clinical and safety-critical settings, professionals and researchers must be able to understand and trust the reasoning behind model predictions. However, the inherently high dimensionality and strong collinearity of spectroscopy data pose a fundamental challenge to model explainability. These properties not only complicate model training but also undermine the stability and consistency of explanations, leading to fluctuations in feature importance across repeated training runs. Feature extraction techniques have been used to reduce the input dimensionality; these new features hinder the connection between the prediction and the original signal. This study proposes SHAPCA, an explainable machine learning pipeline that combines Principal Component Analysis (for dimensionality reduction) and Shapely Additive exPlanations (for post hoc explanation) to provide explanations in the original input space, which a practitioner can interpret and link back to the biological components. The proposed framework enables analysis from both global and local perspectives, revealing the spectral bands that drive overall model behaviour as well as the instance-specific features that influence individual predictions. Numerical analysis demonstrated the interpretability of the results and greater consistency across different runs.

SHAPCA: Consistent and Interpretable Explanations for Machine Learning Models on Spectroscopy Data

Abstract

In recent years, machine learning models have been increasingly applied to spectroscopic datasets for chemical and biomedical analysis. For their successful adoption, particularly in clinical and safety-critical settings, professionals and researchers must be able to understand and trust the reasoning behind model predictions. However, the inherently high dimensionality and strong collinearity of spectroscopy data pose a fundamental challenge to model explainability. These properties not only complicate model training but also undermine the stability and consistency of explanations, leading to fluctuations in feature importance across repeated training runs. Feature extraction techniques have been used to reduce the input dimensionality; these new features hinder the connection between the prediction and the original signal. This study proposes SHAPCA, an explainable machine learning pipeline that combines Principal Component Analysis (for dimensionality reduction) and Shapely Additive exPlanations (for post hoc explanation) to provide explanations in the original input space, which a practitioner can interpret and link back to the biological components. The proposed framework enables analysis from both global and local perspectives, revealing the spectral bands that drive overall model behaviour as well as the instance-specific features that influence individual predictions. Numerical analysis demonstrated the interpretability of the results and greater consistency across different runs.
Paper Structure (29 sections, 10 equations, 6 figures, 5 tables, 2 algorithms)

This paper contains 29 sections, 10 equations, 6 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Literature Review
  3. XAI for spectroscopy
  4. PCA for high-dimensional data
  5. Methodology
  6. Data Preparation
  7. Raman Spectroscopy Dataset.
  8. DRS Dataset.
  9. Pipeline
  10. Sparse PCA
  11. Classification
  12. Train/test configuration and hyperparameters selection
  13. SHAP
  14. Visualization
  15. Global Explanation
  16. ...and 14 more sections

Figures (6)

  • Figure 1: Step-by-step workflow of the proposed interpretable spectroscopic pipeline.
  • Figure 2: A Raman spectroscopy sample. Opacity indicates the most decisive spectral regions for the prediction, while colour encodes intensity relative to the dataset average (red = higher, blue = lower).
  • Figure 3: Global explanations for the binary Raman spectroscopy task (classes H and PML). The grey curves show the class-mean spectra.
  • Figure 4: Local explanation for a binary classification instance correctly predicted as PML. The grey curve represents the Raman spectrum of the instance.
  • Figure 5: Global explanation of DRS multiclass classification across all classes. The grey curve shows the mean spectrum of each class.
  • ...and 1 more figures