Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?

TL;DR

This work tackles the challenge of learning reward functions from human preferences in RLHF with an emphasis on interpretability. It introduces end-to-end differentiable decision trees (DDTs) that split reward prediction into interpretable routing decisions, with two leaf formulations: CRL and IL. Through CartPole, MNIST Gridworlds, and Atari, the approach demonstrates that IL leaves often yield interpretable, visually explainable rewards with competitive RL performance, while CRL can struggle in high-dimensional tasks; the framework also enables detection of silent misalignment before RL runs. The results highlight a trade-off between reward shaping and interpretability, and position reward DDTs as a practical alignment-debugger tool for evaluating and refining human-aligned reward functions in complex domains.

Abstract

Reinforcement Learning from Human Feedback (RLHF) has emerged as a popular paradigm for capturing human intent to alleviate the challenges of hand-crafting the reward values. Despite the increasing interest in RLHF, most works learn black box reward functions that while expressive are difficult to interpret and often require running the whole costly process of RL before we can even decipher if these frameworks are actually aligned with human preferences. We propose and evaluate a novel approach for learning expressive and interpretable reward functions from preferences using Differentiable Decision Trees (DDTs). Our experiments across several domains, including CartPole, Visual Gridworld environments and Atari games, provide evidence that the tree structure of our learned reward function is useful in determining the extent to which the reward function is aligned with human preferences. We also provide experimental evidence that not only shows that reward DDTs can often achieve competitive RL performance when compared with larger capacity deep neural network reward functions but also demonstrates the diagnostic utility of our framework in checking alignment of learned reward functions. We also observe that the choice between soft and hard (argmax) output of reward DDT reveals a tension between wanting highly shaped rewards to ensure good RL performance, while also wanting simpler, more interpretable rewards. Videos and code, are available at: https://sites.google.com/view/ddt-rlhf

Figures (13)

• Figure 1: We propose an end-to-end differentiable approach for training reward functions using differentiable decision trees via trajectory preference labels to enable interpretability and identification of misalignment of the learned reward function.
• Figure 2: Routing probability of an internal node in a DDT.
• Figure 3: Identifying Misalignment in the CartPole Reward DDT. The heatmap for each internal node depicts the learned routing probability. Leaf nodes are depicted as circular nodes with their soft reward values. The tree learns that small magnitude pole angles are good and should be routed to a +1 reward but there is no learned decision boundary that clearly captures the preference that cart position stay within the range $[-2.4,2.4]$ showing that learned reward is misaligned due to the bias in the training dataset---the cartpole usually falls over long before the cart runs off the track.
• Figure 4: MNIST Gridworld with a pair of trajectories where the blue trajectory is preferred.
• Figure 5: Interpreting the MNIST (0-3) Interpolated Leaf Reward DDT. Leaf nodes are depicted as circular nodes with their learned reward values. (a) Visualization of activation maps give insight into the learned routing of the DDT. (b) Visualization of synthetic traces along with their respective routing probabilities.