Textual Explanations and Their Evaluations for Reinforcement Learning Policy

TL;DR

The paper tackles the challenge of making reinforcement learning policies interpretable by introducing a text-to-rule XRL framework that generates textual explanations, converts them into transparent rules, and evaluates their fidelity and practical performance. The approach combines an LLM-based explainer interface, a clustering-based summarizer, and predicate-driven discretization to produce concise, frequent-state explanations, followed by a rule-extraction and refinement process. Key contributions include a Gini-based predicate thresholding method, a min-duplicate and max-F1 refinement strategy, and a comprehensive evaluation across open-source RL environments and a telecom use case, showing improved explanation fidelity and potential for deploying transparent alternatives to black-box policies. The framework supports interactive querying, systematic evaluation, and the potential to replace or augment black-box agents with interpretable rules under acceptable performance trade-offs, advancing the XRL field for real-world applications.

Abstract

Understanding a Reinforcement Learning (RL) policy is crucial for ensuring that autonomous agents behave according to human expectations. This goal can be achieved using Explainable Reinforcement Learning (XRL) techniques. Although textual explanations are easily understood by humans, ensuring their correctness remains a challenge, and evaluations in state-of-the-art remain limited. We present a novel XRL framework for generating textual explanations, converting them into a set of transparent rules, improving their quality, and evaluating them. Expert's knowledge can be incorporated into this framework, and an automatic predicate generator is also proposed to determine the semantic information of a state. Textual explanations are generated using a Large Language Model (LLM) and a clustering technique to identify frequent conditions. These conditions are then converted into rules to evaluate their properties, fidelity, and performance in the deployed environment. Two refinement techniques are proposed to improve the quality of explanations and reduce conflicting information. Experiments were conducted in three open-source environments to enable reproducibility, and in a telecom use case to evaluate the industrial applicability of the proposed XRL framework. This framework addresses the limitations of an existing method, Autonomous Policy Explanation, and the generated transparent rules can achieve satisfactory performance on certain tasks. This framework also enables a systematic and quantitative evaluation of textual explanations, providing valuable insights for the XRL field.

Figures (3)

• Figure 1: Overall framework of the proposed method, where the top half illustrates the explanation generation process, while the shaded area represents evaluation using rule extraction.
• Figure 2: Layout of base stations used in use case.
• Figure 3: Partially generated textual explanations.