CrystalBox: Future-Based Explanations for Input-Driven Deep RL Systems

TL;DR

CrystalBox is presented, a novel, model-agnostic, posthoc explainability framework for Deep Reinforcement Learning (DRL) controllers in the large family of input-driven environments which includes computer systems, and its capability to generate high-fidelity explanations is demonstrated.

Abstract

We present CrystalBox, a novel, model-agnostic, posthoc explainability framework for Deep Reinforcement Learning (DRL) controllers in the large family of input-driven environments which includes computer systems. We combine the natural decomposability of reward functions in input-driven environments with the explanatory power of decomposed returns. We propose an efficient algorithm to generate future-based explanations across both discrete and continuous control environments. Using applications such as adaptive bitrate streaming and congestion control, we demonstrate CrystalBox's capability to generate high-fidelity explanations. We further illustrate its higher utility across three practical use cases: contrastive explanations, network observability, and guided reward design, as opposed to prior explainability techniques that identify salient features.

Figures (14)

• Figure 1: LIMEribeiro2016should's explanation for the motivating states. We observe that in both actions and in both states, LIME presents a similar explanation: recent transmission times, chunk sizes, and buffer are top features.
• Figure 2: CrystalBox's explanation for the two motivating states in Section \ref{['sec:motivation']}. CrystalBox allows us to understand why the controller's actions are appropriate in both states by letting us compare their decomposed future returns to those of alternatives.
• Figure 3: System Diagram of CrystalBox: CrystalBox consists of two components: a learned decomposed returns predictor and a post-processing module. We train a function approximator once to predict the decomposed returns by (i) collecting rollouts of the policy, pre-processing the rollouts to form a dataset, and (ii) employing supervised learning. Once trained, we give the query state and action to this approximator, obtain its predicted decomposed returns, and optionally post-process them.
• Figure 4: Fidelity Evaluation of CrystalBox: Distribution of Squared Error of different methods to the ground truth samples in ABR and CC. CrystalBox offers predictions with the lowest error to the ground truth in all three return components of both environments, for both factual and counterfactual actions. Note that the values of all the returns are scaled to the range [0, 1] before being measured for error. The y-axis in results for ABR is adjusted due to the inherent tail-ended nature of ABR.
• Figure 6: Large Performance Drop Event Detection: We analyze the efficacy of different predictors for detecting large performance drops. We identify events happening by detecting if samples of the ground-truth return exceed a threshold.