Tuning for Trustworthiness -- Balancing Performance and Explanation Consistency in Neural Network Optimization

TL;DR

The paper tackles the problem of balancing predictive performance with explanation reliability by introducing XAI consistency as a formal objective in hyperparameter tuning. It defines three consistency metrics and integrates them into a SPOT-based multi-objective framework, using equal-weighting and desirability-based strategies to guide model selection. Through experiments on California Housing, it shows that multi-objective tuning yields near-perfect attribution agreement across methods, while single-objective tuning can sacrifice interpretability; a trade-off zone with high consistency and competitive accuracy emerges as particularly promising. The findings suggest that models tuned for consistency may generalize better and be more robust, highlighting a path toward trustworthiness in high-stakes AI applications by aligning optimization with both performance and explanation stability.

Abstract

Despite the growing interest in Explainable Artificial Intelligence (XAI), explainability is rarely considered during hyperparameter tuning or neural architecture optimization, where the focus remains primarily on minimizing predictive loss. In this work, we introduce the novel concept of XAI consistency, defined as the agreement among different feature attribution methods, and propose new metrics to quantify it. For the first time, we integrate XAI consistency directly into the hyperparameter tuning objective, creating a multi-objective optimization framework that balances predictive performance with explanation robustness. Implemented within the Sequential Parameter Optimization Toolbox (SPOT), our approach uses both weighted aggregation and desirability-based strategies to guide model selection. Through our proposed framework and supporting tools, we explore the impact of incorporating XAI consistency into the optimization process. This enables us to characterize distinct regions in the architecture configuration space: one region with poor performance and comparatively low interpretability, another with strong predictive performance but weak interpretability due to low \gls{xai} consistency, and a trade-off region that balances both objectives by offering high interpretability alongside competitive performance. Beyond introducing this novel approach, our research provides a foundation for future investigations into whether models from the trade-off zone-balancing performance loss and XAI consistency-exhibit greater robustness by avoiding overfitting to training performance, thereby leading to more reliable predictions on out-of-distribution data.

Figures (6)

• Figure 1: Workflow of the surrogate model-based optimization algorithm in . The process alternates between building surrogate models and optimizing them, with new candidate solutions evaluated on the true objectives and added to the dataset until a stopping criterion is met.
• Figure 2: Desirability functions used for tuning: (Left) Desirability of the performance loss. Values $\leq$ 0.1 achieve the highest desirability, while values $\geq$ 0.7 are considered undesirable. (Right) Desirability of the negative XAI consistency loss. Values are negated due to the spotpython minimization setup. A value of --1 achieves the highest desirability, while values $\geq$ --0.5 are considered undesirable.
• Figure 3: Pareto plot of a Latin hypercube sampling of 100 points in the design space. The y-axis represents XAI consistency values, while the x-axis shows the MSE values for different tuning configurations. The Pareto front and its corresponding points are marked in black, while the remaining configurations are shown as grey points. Configurations with MSE values above 3 are excluded to improve visualization of the relevant regions, resulting in 94 remaining points in the design space.
• Figure 4: Feature attribution values computed using different XAI methods on the validation dataset, after tuning the network architecture based solely on performance loss (single-objective tuning).
• Figure 5: Feature attribution values computed using different XAI methods on the validation dataset, after tuning the network architecture with equal weighting of XAI consistency and performance loss (multi-objective weighted approach).