Programmatically Interpretable Reinforcement Learning

TL;DR

<3-5 sentence high-level summary> PIRL introduces a framework for learning policies as interpretable programs within a domain-specific language, enabling formal verification. It presents Neurally Directed Program Search (NDPS), which guides local search in the program space using a neural policy oracle and augments search with input-driven exploration. Empirical evaluation in Torcs and classic control tasks shows that PIRL yields human-readable policies with smoother trajectories, competitive performance, and superior transfer to unseen environments, along with verifiable properties achieved through symbolic methods. The work highlights the trade-offs between interpretability and performance and outlines directions for handling perceptual inputs and stochastic policies.

Abstract

We present a reinforcement learning framework, called Programmatically Interpretable Reinforcement Learning (PIRL), that is designed to generate interpretable and verifiable agent policies. Unlike the popular Deep Reinforcement Learning (DRL) paradigm, which represents policies by neural networks, PIRL represents policies using a high-level, domain-specific programming language. Such programmatic policies have the benefits of being more easily interpreted than neural networks, and being amenable to verification by symbolic methods. We propose a new method, called Neurally Directed Program Search (NDPS), for solving the challenging nonsmooth optimization problem of finding a programmatic policy with maximal reward. NDPS works by first learning a neural policy network using DRL, and then performing a local search over programmatic policies that seeks to minimize a distance from this neural "oracle". We evaluate NDPS on the task of learning to drive a simulated car in the TORCS car-racing environment. We demonstrate that NDPS is able to discover human-readable policies that pass some significant performance bars. We also show that PIRL policies can have smoother trajectories, and can be more easily transferred to environments not encountered during training, than corresponding policies discovered by DRL.

Figures (6)

• Figure 1: Syntax of the policy language.
• Figure 2: A programmatic policy for acceleration, automatically discovered by the Ndps algorithm. $h_\mathtt{RPM}$ and $h_\mathtt{TrackPos}$ represent histories for the $\mathtt{RPM}$ and $\mathtt{TrackPos}$ sensors, respectively.
• Figure 3: Slice of steering actions taken by the DRL and Ndps agents, during the CG-Speedway-1 race. This figure demonstrates that the Ndps agent drives more smoothly.
• Figure 4: Distance raced by the agents as the block probability increases for a particular sensor(s) on Aalborg. The Ndps agent is more robust to blocked sensors.
• Figure 5: A programmatic policy for steering, automatically discovered by the Ndps algorithm with training on Aalborg.