4Hammer: a board-game reinforcement learning environment for the hour long time frame

TL;DR

4Hammer tackles the challenge of long-horizon decision making by providing a digital reinforcement learning environment that emulates Warhammer 40,000 Combat Patrol. Built on the Rulebook DSL and a Godot-based graphical engine, it supports both headless RL and graphical interaction to facilitate LLM-based state understanding and traditional reinforcement learning. The architecture centers on modular libraries (Stats, Board, Rules) with serialization into textual, binary, and tensor formats, and includes drivers and outputs for integration with ML pipelines. Preliminary experiments validate robustness and demonstrate RL (PPO) in headless mode and graphical-driver testing with Gemini 2 Flash, establishing a testbed for hour-long, text-rich board-game AI research and LLM evaluation.

Abstract

Large Language Models (LLMs) have demonstrated strong performance on tasks with short time frames, but struggle with tasks requiring longer durations. While datasets covering extended-duration tasks, such as software engineering tasks or video games, do exist, there are currently few implementations of complex board games specifically designed for reinforcement learning and LLM evaluation. To address this gap, we propose the 4Hammer reinforcement learning environment, a digital twin simulation of a subset of Warhammer 40,000-a complex, zero-sum board game. Warhammer 40,000 features intricate rules, requiring human players to thoroughly read and understand over 50 pages of detailed natural language rules, grasp the interactions between their game pieces and those of their opponents, and independently track and communicate the evolving game state.

Figures (5)

• Figure 1: Hierarchical representation of the components in 4Hammer. Purple components are implemented exclusively in Rulebook, while yellow components represent Python source files. The component labeled 4hammer_godot_project comprises Godot engine files responsible for graphical elements. Cyan-colored nodes represent binary objects such as executables and libraries. Dashed edges indicate input-output relationships, illustrating inputs processed by tools (rlc or the Godot editor) to produce outputs. Solid lines represent composition, illustrating how various artifacts are combined to produce new components. Dotted lines indicate network communication occurring at runtime.
• Figure 2: The single_attack game sequence (left) and the engine‐level skirmish (right).
• Figure 3: Average score obtained by the player 0 while training on single_shooting_maximize.rl
• Figure 4: Average score obtained by the agent during training on single_turn.rl.
• Figure 5: Gemini 2 Flash inner monologue when considering if it should try rerolling a pair of dices. The decision required the model to understand which pieces belonged to the opponent and which did not, what was the decision to perform in the moment, the previous result that was being prompted to reroll, what was the cost of rerolling, and if that cost was worth it or not.