Counterfactual Explanations on Robust Perceptual Geodesics

TL;DR

This work addresses the difficulty of producing semantically meaningful counterfactual explanations in high-dimensional vision by reframing counterfactual generation as geodesic traversal on a latent manifold. It introduces Perceptual Counterfactual Geodesics (PCG), which constructs a robust perceptual metric $G_R$ from multiple layers of robust vision features and pulls it back to latent space as $G_Z$, yielding geometry-aware paths that stay on-manifold and evolve semantically. The authors implement a two-phase optimization: Phase 1 learns a robust geodesic between the input and a target exemplar, and Phase 2 jointly refines the endpoint under a classification loss with re-anchoring, producing minimal, faithful counterfactuals. Across AFHQ, FFHQ, and PlantVillage, PCG outperforms baselines on geometry-aware metrics (e.g., $\mathcal{L}_{\mathcal{R}}$, SM, MAS) and reveals failure modes of traditional latent-space counterfactual methods, highlighting the importance of robust latent geometry for trustworthy explanations.

Abstract

Latent-space optimization methods for counterfactual explanations - framed as minimal semantic perturbations that change model predictions - inherit the ambiguity of Wachter et al.'s objective: the choice of distance metric dictates whether perturbations are meaningful or adversarial. Existing approaches adopt flat or misaligned geometries, leading to off-manifold artifacts, semantic drift, or adversarial collapse. We introduce Perceptual Counterfactual Geodesics (PCG), a method that constructs counterfactuals by tracing geodesics under a perceptually Riemannian metric induced from robust vision features. This geometry aligns with human perception and penalizes brittle directions, enabling smooth, on-manifold, semantically valid transitions. Experiments on three vision datasets show that PCG outperforms baselines and reveals failure modes hidden under standard metrics.

Figures (13)

• Figure 1: Schematic of PCG. An input is mapped through an encoder-generator pair. A linear latent path to a perceptually plausible target-class sample (Class B, brown region) is refined in Phase 1 into the blue geodesic by minimizing robust perceptual energy. In Phase 2, the endpoint and intermediate points are jointly optimized under classification loss and robust energy, resulting in the red counterfactual geodesic. The green trajectory (REVISE, VSGD) ignores manifold geometry, strays off-manifold and produces off-manifold AEs. The yellow trajectory (RSGD/-C) conforms to a fragile geometry, getting stuck in on-manifold adversarial regions (Class B, outside brown region).
• Figure 2: Interpolation paths under four latent geometries based on StyleGAN2 (top→bottom). (a) $Z$-linear (Euclidean): flat latent metric; off-manifold artifacts. (b) Pixel MSE pullback: Euclidean metric pulled back to $Z$; brittle, incoherent paths. (c) Standard feature pullback: non-robust ResNet-50; better semantics but still fading and discontinuities. (d) Robust perceptual pullback (ours): robust ResNet-50; smooth, consistent, on-manifold trajectories. See Appendix \ref{['app:interp_more']} for StyleGAN3 results.
• Figure 3: Perceptual Counterfactual Geodesics. Rows 1 and 3: initial geodesics from Phase 1 between an input and a target-class sample. Rows 2 and 4: counterfactual geodesics after Phase 2, where the endpoint is optimized with the path. from Phase 2 stay in robust regions of the manifold and preserve semantic continuity. Results from StyleGAN2 (see Appendix \ref{['app:pcg_more']} for StyleGAN3)
• Figure 4: Qualitative comparison of counterfactuals across methods with StyleGAN2. Columns show input images followed by counterfactuals from PCG (ours), RSGD, RSGD-C, VSGD, and REVISE. Rows indicate input and target /class. PCG produces minimal, semantically faithful changes along robust geodesics, while baselines often show off-manifold artifacts, semantic drift, or adversarial collapse. Optimization details for baselines are presented in Appendix \ref{['app:baselines-opt']}.
• Figure 5: Interpolations on FFHQ under four geometries. Rows (top to bottom): $Z$-lerp, $\mathcal{X}_{\text{MSE}}$ pullback, $\mathcal{F}_{\text{MSE}}$ pullback, and robust $\mathcal{R}_{\text{MSE}}$ pullback. The robust row shows a smooth, semantically consistent evolution (e.g., gradual attribute change without identity drift), whereas the other geometries introduce off-manifold blends and texture/illumination artifacts mid-path.