EvoXplain: When Machine Learning Models Agree on Predictions but Disagree on Why -- Measuring Mechanistic Multiplicity Across Training Runs

TL;DR

EvoXplain introduces a diagnostic framework to quantify the stability of model explanations across repeated training, treating attribution patterns as samples from the stochastic optimisation process rather than fixed model properties. By clustering SHAP-based explanations from many retrainings, it reveals whether a model class converges to a single explanatory mode or exhibits multiple, well-separated explanatory basins, quantified with mechanistic entropy. Applied to Breast Cancer and COMPAS datasets across Logistic Regression and Random Forests, EvoXplain uncovers robust explanation multiplicity even at high predictive accuracy, and shows that averaging explanations can obscure distinct explanatory stories. The work argues for interpretability to be assessed as a population-level property over retrained instances, with implications for auditing, governance, and responsible AI. It also clarifies limitations and calls for broader validation across modalities and explanation paradigms, including potential extensions to large-scale foundation models.

Abstract

Machine learning models are primarily judged by predictive performance, especially in applied settings. Once a model reaches high accuracy, its explanation is often assumed to be correct and trustworthy. However, this assumption raises an overlooked question: when two models achieve high accuracy, do they rely on the same internal logic, or do they reach the same outcome via different -- and potentially competing -- mechanisms? We introduce EvoXplain, a diagnostic framework that measures the stability of model explanations across repeated training. Rather than analysing a single trained model, EvoXplain treats explanations as samples drawn from the stochastic optimisation process itself -- without aggregating predictions or constructing ensembles -- and examines whether these samples form a single coherent explanation or separate into multiple, distinct explanatory modes. We evaluate EvoXplain on the Breast Cancer and COMPAS datasets using two widely deployed model classes: Logistic Regression and Random Forests. Although all models achieve high predictive accuracy, their explanations frequently exhibit clear multimodality. Even models commonly assumed to be stable, such as Logistic Regression, can produce multiple well-separated explanatory basins under repeated training on the same data split. These differences are not explained by hyperparameter variation or simple performance trade-offs. EvoXplain does not attempt to select a 'correct' explanation. Instead, it makes explanatory instability visible and quantifiable, revealing when single-instance or averaged explanations obscure the existence of multiple underlying mechanisms. More broadly, EvoXplain reframes interpretability as a property of a model class under repeated instantiation, rather than of any single trained model.

Figures (7)

• Figure 1: Breast Cancer — Logistic Regression. Left: PCA embedding of explanation (attribution) vectors obtained from repeated training runs on the same train–test split. Points are coloured by cluster assignment identified in explanation space. Red stars indicate representative cluster centroids, while the black cross denotes the average attribution vector across all runs. Right: The same embedding with predictive accuracy overlaid. Accuracy remains high across the full extent of the explanation manifold, including across distinct clusters, indicating that multiple explanatory modes coexist at comparable performance.
• Figure 2: Breast Cancer - Logistic Regression: universal explanation manifold. PCA embedding of explanation (SHAP) vectors obtained by stacking runs across all train-test splits. Each point corresponds to a trained model instance and is coloured by predictive accuracy. Multiple persistent explanation tracks are visible across splits, indicating that explanation variability is not confined to a single resampling of the data. The black $\times$ marks the global average attribution vector, computed as the arithmetic mean of all SHAP vectors across runs and clusters; this point does not correspond to a typical location on the populated manifold.
• Figure 3: COMPAS — Logistic Regression (single split). Left: PCA embedding of explanation (SHAP) vectors obtained from repeated training runs on a fixed train–test split (split 105). Points are coloured by explanation cluster assignment. Red stars indicate representative cluster centroids, and the black $\times$ denotes the average attribution vector across all runs. Right: The same embedding with predictive accuracy overlaid. Accuracy remains within a narrow range across the explanation manifold, indicating that distinct explanatory modes coexist at similar predictive performance.
• Figure 4: COMPAS - Logistic Regression: universal explanation manifold. PCA embedding of explanation (SHAP) vectors obtained by stacking model runs across all train-test splits. Each point corresponds to a trained model instance and is coloured by predictive accuracy. Multiple separated explanation tracks persist across splits, indicating that explanation variability is not confined to a single resampling of the data. The black $\times$ marks the global average attribution vector, computed as the arithmetic mean of all SHAP vectors across runs and clusters; this point lies away from the populated tracks of the manifold.
• Figure 5: Centroid feature profiles for explanatory basins (Breast Cancer, Logistic Regression). Mean absolute SHAP values for the top-five features of each explanation cluster identified in split 104 under varied regularisation. Cluster 0 (C$\approx$0.32, n=556, accuracy = 0.971) emphasises macroscopic size and surface descriptors (e.g. worst radius, worst perimeter, worst texture), while Cluster 1 (C$\approx$24.1, n=444, accuracy = 0.963) places substantially greater weight on variability- and complexity-related features (e.g. fractal dimension error, area error, mean concavity). Feature magnitudes differ by more than an order of magnitude across clusters, indicating qualitatively distinct explanatory mechanisms rather than small perturbations of a single solution, despite near-identical predictive accuracy.