Bridging Interpretability and Optimization: Provably Attribution-Weighted Actor-Critic in Reproducing Kernel Hilbert Spaces
Na Li, Hangguan Shan, Wei Ni, Wenjie Zhang, Xinyu Li
TL;DR
The paper tackles the challenge of interpreting and stabilizing policy updates in actor–critic reinforcement learning by introducing RSA2C, an attribution‑aware, kernelized AC that embeds the Actor and two Critics in RKHS and computes state attributions online via RKHS–SHAP. Attributions are converted into adaptive Mahalanobis weights through a vector‑valued operator‑valued kernel, influencing Actor gradients and Advantage Critic targets to yield more stable, efficient learning. The authors provide a global, non‑asymptotic convergence guarantee under state perturbations, decomposing the learning gap into attribution‑driven perturbation and convergence terms. Empirically, RSA2C demonstrates improved efficiency, stability, and intrinsic interpretability across standard continuous control tasks, with CME‑based attributions offering more robust behavior under noise and better feature‑level explanations than KME. The work contributes a principled, scalable framework that integrates dimension‑level explanations directly into the learning loop, with potential extensions to high‑dimensional or pixel‑based settings using scalable kernel approximations.
Abstract
Actor-critic (AC) methods are a cornerstone of reinforcement learning (RL) but offer limited interpretability. Current explainable RL methods seldom use state attributions to assist training. Rather, they treat all state features equally, thereby neglecting the heterogeneous impacts of individual state dimensions on the reward. We propose RKHS--SHAP-based Advanced Actor--Critic (RSA2C), an attribution-aware, kernelized, two-timescale AC algorithm, including Actor, Value Critic, and Advantage Critic. The Actor is instantiated in a vector-valued reproducing kernel Hilbert space (RKHS) with a Mahalanobis-weighted operator-valued kernel, while the Value Critic and Advantage Critic reside in scalar RKHSs. These RKHS-enhanced components use sparsified dictionaries: the Value Critic maintains its own dictionary, while the Actor and Advantage Critic share one. State attributions, computed from the Value Critic via RKHS--SHAP (kernel mean embedding for on-manifold expectations and conditional mean embedding for off-manifold expectations), are converted into Mahalanobis-gated weights that modulate Actor gradients and Advantage Critic targets. Theoretically, we derive a global, non-asymptotic convergence bound under state perturbations, showing stability through the perturbation-error term and efficiency through the convergence-error term. Empirical results on three standard continuous-control environments show that our algorithm achieves efficiency, stability, and interpretability.