Benchmarking Counterfactual Interpretability in Deep Learning Models for Time Series Classification

TL;DR

This work addresses the lack of standardized benchmarks for counterfactual explanations in time series classification by redesigning evaluation metrics and conducting a comprehensive benchmark of 6 CF methods across 20 univariate and 10 multivariate datasets using 3 classifiers. It introduces a full metric suite—Validity, Proximity ($L_1$, $L_2$, $L_{\,inf}$), Sparsity and Sensitivity (ThreshL0, Sens), Segment sparsity, Plausibility (Dist_all, Dist_class via latent representations), Generation Time, and Consistency—to capture CF quality, including a new Consistency dimension. The results show no single method dominates across metrics or datasets; classifier choice significantly influences CF performance, and results vary between univariate and multivariate settings. The paper provides case studies and practical guidelines to inform method selection and emphasize robustness, efficiency, and interpretability in real-world deployments.

Abstract

The popularity of deep learning methods in the time series domain boosts interest in interpretability studies, including counterfactual (CF) methods. CF methods identify minimal changes in instances to alter the model predictions. Despite extensive research, no existing work benchmarks CF methods in the time series domain. Additionally, the results reported in the literature are inconclusive due to the limited number of datasets and inadequate metrics. In this work, we redesign quantitative metrics to accurately capture desirable characteristics in CFs. We specifically redesign the metrics for sparsity and plausibility and introduce a new metric for consistency. Combined with validity, generation time, and proximity, we form a comprehensive metric set. We systematically benchmark 6 different CF methods on 20 univariate datasets and 10 multivariate datasets with 3 different classifiers. Results indicate that the performance of CF methods varies across metrics and among different models. Finally, we provide case studies and a guideline for practical usage.

Figures (27)

• Figure 1: An example CF from the GunPoint dataset whose $L_0$ norm is 1.00 contradicting human perception i.e., all the time steps are changed.
• Figure 2: Average ranking of metrics for models on 20 Univariate datasets.
• Figure 3: Average ranking of metrics for models on 10 Multivariate datasets.
• Figure 4: Visualization CF results on GunPoint dataset instance 105.
• Figure 5: Critical difference diagram of $Gtime$ on 20 Univariate datasets.