Rulebook: bringing co-routines to reinforcement learning environments

TL;DR

This paper addresses the rigidity of RL environment interfaces by proposing Rulebook, a co-routine-based DSL compiled via MLIR that automatically generates a state machine to interact with ML components without incurring runtime overhead. It establishes design goals including main-loop independence, inspectability, serializability, checkability, and C compatibility, enabling environments that are easily inspectable and serializable while remaining fast. Empirically, Rulebook re-implements six OpenSpiel board games with performance competitive to unoptimized C++ baselines and demonstrates a scalable Warhammer 40,000 case study, illustrating practical viability for complex games and RL data tooling. The work highlights static analysis advantages of coroutine-based environments and outlines future directions such as enabling mutually recursive actions and broader tooling support for RL datasets.

Abstract

Reinforcement learning (RL) algorithms, due to their reliance on external systems to learn from, require digital environments (e.g., simulators) with very simple interfaces, which in turn constrain significantly the implementation of such environments. In particular, these environments are implemented either as separate processes or as state machines, leading to synchronization and communication overheads in the first case, and to unstructured programming in the second. We propose a new domain-specific, co-routine-based, compiled language, called Rulebook, designed to automatically generate the state machine required to interact with machine learning (ML) algorithms and similar applications, with no performance overhead. Rulebook allows users to express programs without needing to be aware of the specific interface required by the ML components. By decoupling the execution model of the program from the syntactical encoding of the program, and thus without the need for manual state management, Rulebook allows to create larger and more sophisticated environments at a lower development cost.

Figures (6)

• Figure 1: rlc toolchain and generated tools
• Figure 2: Action graphs
• Figure : gymTic-Tac-Toe wrapper example
• Figure : Class generated by the inspection of the tic tac toe action declaration
• Figure : Example of translated Rulebook action statement from listing \ref{['lst:tictactoecls']}