EXPIL: Explanatory Predicate Invention for Learning in Games

TL;DR

EXPIL addresses the interpretability bottleneck in reinforcement learning by automatically discovering explanatory predicates from a pretrained agent's replay buffer and constructing first-order-logic policies. It introduces Necessity and Sufficiency predicates to evaluate and refine these concepts, using beam search to form weighted policy clauses and actor-critic updates to train strategy. Across Getout, Loot, and Threefish, EXPIL matches or surpasses neural PPO and the state-of-the-art NeSy baseline while requiring minimal hand-crafted priors, enabling explainable behavior in relational environments. This work advances interpretable, robust RL by reducing the dependency on predefined background knowledge and providing a scalable framework for explanatory predicate invention in learning-from-games settings.

Abstract

Reinforcement learning (RL) has proven to be a powerful tool for training agents that excel in various games. However, the black-box nature of neural network models often hinders our ability to understand the reasoning behind the agent's actions. Recent research has attempted to address this issue by using the guidance of pretrained neural agents to encode logic-based policies, allowing for interpretable decisions. A drawback of such approaches is the requirement of large amounts of predefined background knowledge in the form of predicates, limiting its applicability and scalability. In this work, we propose a novel approach, Explanatory Predicate Invention for Learning in Games (EXPIL), that identifies and extracts predicates from a pretrained neural agent, later used in the logic-based agents, reducing the dependency on predefined background knowledge. Our experimental evaluation on various games demonstrate the effectiveness of EXPIL in achieving explainable behavior in logic agents while requiring less background knowledge.

Figures (8)

• Figure 1: EXPIL introduces predicate invention within neuro-symbolic RL agents. EXPIL extracts concepts from a replay buffer, lated employed to compute optimal actions through neuro-symbolic policies. In contrast to neural policies, EXPIL generates highly interpretable policies using logic and requires few hand-crafted priors compared to conventional neuro-symbolic policies.
• Figure 2: EXPIL Architecture. Top: EXPIL uses a state/action game buffer of pretrained agents to extract logical states $\mathcal{S}$, invents necessity predicates $\mathcal{P}_\mathit{ness}$ and sufficiency predicates $\mathcal{P}_\mathit{suff}$, deduces policy clauses $\mathcal{C}$, and finally learns an optimized policy $\pi_\mathcal{C}$ through interaction with the game environment. Bottom: At inference time, the logic agent uses the optimized policy and game states as input, evaluates valid policy clauses from the environment, and selects the action corresponding to the rule with the highest evaluation.
• Figure 3: The Predicate Invention module of EXPIL. EXPIL utilizes an object-centric state-action buffer as input for predicate invention and rule reasoning. Candidate predicates are invented to serve and combined within clauses of the policy. The reasoned policy clauses are evaluated, and promising rules are selected as output to the logical agent.
• Figure 4: An example step for sufficiency predicate invention. In each rectangle box: the $8$ circles above the dashed line indicate the states taking the action $\texttt{Jump}$, while the remaining circles indicate the states taking other actions. Boxes $1, 2, 3$ shows the evaluation results of three different predicates, with blue circles of box $i\in[1,2,3]$ representing the positive states that satisfy the predicate $i$ and purple circles of box $i\in[1,2,3]$ representing the negative states that satisfy the predicate $i$. The $\mu_p(\mathcal{S}_a^+)$ and $\mu_p(\mathcal{S}_a^-)$ at the bottom indicate the scores of the invented sufficiency predicate.
• Figure 5: Environments used to evaluate EXPIL and baselines.