Uncovering Latent Phase Structures and Branching Logic in Locomotion Policies: A Case Study on HalfCheetah

Abstract

In locomotion control tasks, Deep Reinforcement Learning (DRL) has demonstrated high performance; however, the decision-making process of the learned policy remains a black box, making it difficult for humans to understand. On the other hand, in periodic motions such as walking, it is well known that implicit motion phases exist, such as the stance phase and the swing phase. Focusing on this point, this study hypothesizes that a policy trained for locomotion control may also represent a phase structure that is interpretable by humans. To examine this hypothesis in a controlled setting, we consider a locomotion task that is amenable to observing whether a policy autonomously acquires temporally structured phases through interaction with the environment. To verify this hypothesis, in the MuJoCo locomotion benchmark HalfCheetah-v5, the state transition sequences acquired by a policy trained for walking control through interaction with the environment were aggregated into semantic phases based on state similarity and consistency of subsequent transitions. As a result, we demonstrated that the state sequences generated by the trained policy exhibit periodic phase transition structures as well as phase branching. Furthermore, by approximating the states and actions corresponding to each semantic phase using Explainable Boosting Machines (EBMs), we analyzed phase-dependent decision making-namely, which state features the policy function attends to and how it controls action outputs in each phase. These results suggest that neural network-based policies, which are often regarded as black boxes, can autonomously acquire interpretable phase structures and logical branching mechanisms.

Figures (9)

• Figure 1: Overview of the analysis of a locomotion-control policy function. This analysis is composed of two stages: (Left) identification of temporally semantic phases by embedding state sequences and clustering with transition entropy minimization, and (Right) phase-wise surrogate modeling using an Explainable Boosting Machine (EBM) to reveal state–action contribution rules within each phase.
• Figure 2: Visualization of the state space projected by UMAP. Each dot represents a state, colored by its assigned cluster (phase). The blue edges connect each state to its successor state at the next time step. To clarify the dominant flow of phase transitions, white arrows are overlaid on the edges corresponding to transitions that cumulatively account for more than 70% of observed transitions from each source cluster (see Table 2), indicating the direction of those transitions.
• Figure 3: Randomly sampled rendered states from two phase--transition sequences. The upper row shows examples drawn from the transition cycle $\mathrm{Cluster}\;0 \rightarrow 1 \rightarrow 2 \rightarrow 3 \rightarrow 4 \rightarrow 5 \rightarrow 0$ (Pattern 1), while the lower row shows examples drawn from the longer cycle $\mathrm{Cluster}\;0 \rightarrow 1 \rightarrow 2 \rightarrow 3 \rightarrow 4 \rightarrow 5 \rightarrow 6 \rightarrow 7 \rightarrow 8 \rightarrow 9 \rightarrow 0$ (Pattern 2). For each cluster in the respective sequence, two states were randomly sampled from the corresponding state trajectory and rendered to visualize representative postures along the phase transitions.
• Figure 4: Feature attribution heatmaps for Phase 0 approximated by EBM, together with representative robot postures for each phase. Red frames indicate the state--action pairs whose attribution values exceed 95% of the maximum heatmap value.
• Figure 5: Feature attribution heatmaps for Phase 1, 2 and 3 approximated by EBM, together with representative robot postures for each phase. Red frames indicate the state--action pairs whose attribution values exceed 95% of the maximum heatmap value.