REVEAL-IT: REinforcement learning with Visibility of Evolving Agent poLicy for InTerpretability

TL;DR

The paper addresses the challenge of understanding and improving RL training in complex environments by focusing on the agent's learning process rather than single-step decisions. It introduces REVEAL-IT, which visualizes policy updates as a node-link graph and uses a GNN-based explainer to identify the most impactful subtask-induced updates. The approach couples an online RL data-driven predictor of learning progress with a decomposed explanation graph $G_O = G_X + \Delta G$, enabling curriculum optimization. Experiments in ALFWorld and OpenAI Gym demonstrate improved training efficiency and interpretability, with the GNN explainer effectively mapping important policy updates to agent capabilities.

Abstract

Understanding the agent's learning process, particularly the factors that contribute to its success or failure post-training, is crucial for comprehending the rationale behind the agent's decision-making process. Prior methods clarify the learning process by creating a structural causal model (SCM) or visually representing the distribution of value functions. Nevertheless, these approaches have constraints as they exclusively function in 2D-environments or with uncomplicated transition dynamics. Understanding the agent's learning process in complicated environments or tasks is more challenging. In this paper, we propose REVEAL-IT, a novel framework for explaining the learning process of an agent in complex environments. Initially, we visualize the policy structure and the agent's learning process for various training tasks. By visualizing these findings, we can understand how much a particular training task or stage affects the agent's performance in test. Then, a GNN-based explainer learns to highlight the most important section of the policy, providing a more clear and robust explanation of the agent's learning process. The experiments demonstrate that explanations derived from this framework can effectively help in the optimization of the training tasks, resulting in improved learning efficiency and final performance.

Figures (7)

• Figure 1: The main structure of REVEAL-IT. Assume that we need to train an RL agent within an environment to accomplish complex tasks, while directly training the agent on these tasks is challenging and inefficient. In practical application, we will devise sequences of pre-defined sub-tasks (sub-task 1, sub-task 2,..., sub-task N) for training. In REVEAL-IT, we implement the RL agent in the given environment, allowing it to explore and collect data. Subsequently, we train the controller (the control policy $\pi_\theta$), using the collected data. Then, we visualize the policy updates with a node-link diagram. The visualization will depict the structure of the policy and highlight the specific sections that have been updated. After that, REVEAL-IT employs a GNN-based explanation to examine policy updates and ascertain the significant capabilities that the policy has acquired in a certain sub-task. This will help us comprehend how much a sub-task has improved the agent's performance in the test. Furthermore, the GNN-based explainer provides a clearer and more accurate comprehension of the value of each sub-task in training. This can enhance the effectiveness and efficiency of designing RL training task sequences in real-world settings.
• Figure 2: Visualized important policy updates by GNN explainer. We provide a larger version in Appendix. \ref{['sec:large_ver']}, and detailed analysis in section \ref{['sec:5.3']}. In this figure, the first line depicts the sequential arrangement of sub-tasks that must be accomplished step-by-step to accomplish a given task fully. One of the sub-tasks is located on the left side of the second line. The first to third columns of the tree diagram illustrates the RL policy update process. The blue circles represent nodes in the neural network, while the connections between the circles represent weight updates. Thicker connections indicate larger updates in weight amplitude (selected by GNN explainer). The red circles in the tree diagram on the far right illustrate the specific policy nodes that are active during the evaluation process. The links here represent the revised weights throughout the training phase of this subtask. The orange squares indicate the portions of the policy that are common to several sub-tasks. We opt to depict the 8 interconnected nodes with the most significant weight adjustment, facilitating comprehension of the reinforcement learning policy's learning process and highlighting the policy's shared component more distinctly.
• Figure 3: The distribution of verbs in training tasks. This figure demonstrates how REVEAL-IT optimizes the task sequences. We employ various verbs in tasks to differentiate, as each type necessitates distinct capabilities from the agent. The agent training process reflects the task sequence change from left to right. Analyzing the changes in task distribution shows that the "put" type of task is the most prevalent. As the training advances, the initial focus of training tasks is on teaching the agent to locate and retrieve objects in the environment ("look", "pick"). Subsequently, the agent is trained on tasks that require it to acquire other skills, e.g., "clean", "heat", and "examine". We provide a larger version in Appendix. \ref{['sec:large_ver']}.
• Figure 4: Visualized task examples of ALFWorld ALFWorld20. This benchmark utilizes various household scenarios created within the Ai2Thor environment. All objects can be moved to different locations based on available surfaces and class restrictions in this environment. This allows for the generation of a wide range of new tasks by combining different objects and goal positions in a procedural manner.
• Figure 5: The results of learning process of the GNN explainer.