RLAF: Reinforcement Learning from Automaton Feedback
Mahyar Alinejad, Alvaro Velasquez, Yue Wang, George Atia
TL;DR
The paper introduces RLAF, an automaton-based framework that replaces explicit rewards with DFA-derived trajectory preferences to handle non-Markovian objectives. It decomposes the approach into DFA-based preference elicitation, learned reward functions via pairwise ranking, and static or dynamic policy optimization, with optional transfer of automaton knowledge through Q-value estimates. The authors provide a formal convergence guarantee showing $\\varepsilon$-optimality under standard assumptions and validate the method across discrete and continuous domains, where it outperforms reward-engineering baselines and LTL-guided methods, and demonstrates strong transferability through automaton transitions. This work offers a scalable, human-independent pathway to incorporate temporal logic into RL, enabling efficient learning in tasks with complex sequential constraints and long-horizon dependencies.
Abstract
Reinforcement Learning (RL) in environments with complex, history-dependent reward structures poses significant challenges for traditional methods. In this work, we introduce a novel approach that leverages automaton-based feedback to guide the learning process, replacing explicit reward functions with preferences derived from a deterministic finite automaton (DFA). Unlike conventional approaches that use automata for direct reward specification, our method employs the structure of the DFA to generate preferences over trajectories that are used to learn a reward function, eliminating the need for manual reward engineering. Our framework introduces a static approach that uses the learned reward function directly for policy optimization and a dynamic approach that involves continuous refining of the reward function and policy through iterative updates until convergence. Our experiments in both discrete and continuous environments demonstrate that our approach enables the RL agent to learn effective policies for tasks with temporal dependencies, outperforming traditional reward engineering and automaton-based baselines such as reward machines and LTL-guided methods. Our results highlight the advantages of automaton-based preferences in handling non-Markovian rewards, offering a scalable, efficient, and human-independent alternative to traditional reward modeling. We also provide a convergence guarantee showing that under standard assumptions our automaton-guided preference-based framework learns a policy that is near-optimal with respect to the true non-Markovian objective.