Zero-Shot Instruction Following in RL via Structured LTL Representations
Mattia Giuri, Mathias Jackermeier, Alessandro Abate
TL;DR
This work advances zero-shot instruction following in reinforcement learning by representing complex LTL tasks as sequences of Boolean formulas derived from Büchi automata. The policy is conditioned on these structured formulas via a graph neural network that processes formula graphs and a recurrent module that captures progress along an accepting run, enabling robust generalization when multiple propositions interact. The approach outperforms state-of-the-art baselines on a chess-inspired environment, particularly on infinite-horizon tasks, and demonstrates resilience to increasing task difficulty. The method offers a principled and scalable way to encode structured temporal specifications for generalist RL policies with potential for broader applicability beyond the tested domain.
Abstract
Linear temporal logic (LTL) is a compelling framework for specifying complex, structured tasks for reinforcement learning (RL) agents. Recent work has shown that interpreting LTL instructions as finite automata, which can be seen as high-level programs monitoring task progress, enables learning a single generalist policy capable of executing arbitrary instructions at test time. However, existing approaches fall short in environments where multiple high-level events (i.e., atomic propositions) can be true at the same time and potentially interact in complicated ways. In this work, we propose a novel approach to learning a multi-task policy for following arbitrary LTL instructions that addresses this shortcoming. Our method conditions the policy on sequences of simple Boolean formulae, which directly align with transitions in the automaton, and are encoded via a graph neural network (GNN) to yield structured task representations. Experiments in a complex chess-based environment demonstrate the advantages of our approach.