Scaling Behavior Cloning Improves Causal Reasoning: An Open Model for Real-Time Video Game Playing

TL;DR

This work presents an open, end-to-end pipeline for scaling behavior cloning to real-time cross-game play, including a large annotated dataset (8,300+ hours) and unlabeled data, a novel text-conditioned, autoregressive policy (Pixels2Play) designed for 20 Hz inference on consumer GPUs, and a three-stage pretraining approach using an inverse dynamics model. Through extensive experiments across programmatic and real games, the authors demonstrate that larger models trained on more diverse data achieve lower test loss and stronger causal signal in reasoning, approaching human-level play in several 3D titles. They also show that scaling improves causality and reduces reliance on non-causal cues, providing a practical route to mitigate causal confusion in behavior cloning. The findings have practical impact for deploying generalist, real-time agents on consumer hardware and offer a framework for studying scaling laws and causality in BC across diverse environments.

Abstract

Behavior cloning is enjoying a resurgence in popularity as scaling both model and data sizes proves to provide a strong starting point for many tasks of interest. In this work, we introduce an open recipe for training a video game playing foundation model designed for inference in realtime on a consumer GPU. We release all data (8300+ hours of high quality human gameplay), training and inference code, and pretrained checkpoints under an open license. We show that our best model is capable of playing a variety of 3D video games at a level competitive with human play. We use this recipe to systematically examine the scaling laws of behavior cloning to understand how the model's performance and causal reasoning varies with model and data scale. We first show in a simple toy problem that, for some types of causal reasoning, increasing both the amount of training data and the depth of the network results in the model learning a more causal policy. We then systematically study how causality varies with the number of parameters (and depth) and training steps in scaled models of up to 1.2 billion parameters, and we find similar scaling results to what we observe in the toy problem.

Figures (16)

• Figure 1: Example gameplay sequence with aligned action and text annotations. For visual clarity, we only show the frames where a text annotation is initialized, keyboard actions are simplified to WASD inputs, and mouse clicks are omitted; The highlighted key means the key is pressed, and the arrow indicates mouse movement in the $x$ and $y$ directions.
• Figure 2: (\ref{['subfig:model']}) Architecture of P2P. The core policy transformer and action decoder are both decoder-only transformers. Each timestep begins with a text token $t_i$. Since many frames do not contain a text annotation there is a default text token $t_{null}$ used on these frames. This is followed by image token(s) from video frame $o_{i}$ followed by a learnable “reasoning’’ token $k_{i}$ that grants the model extra computation. The policy transformer then outputs a single action prediction token $a_{in}$. A smaller transformer, the action decoder, then auto-regressively transforms and samples the single action prediction token into the full action space. Then the true action tokens $a_i$ are input so that $a_{in}$ at time $i+1$ can attend to the true action tokens from time $i$. (\ref{['subfig:mask']}) Attention mask used in our transformer policy (green denotes $1$ and gray $0$). This custom mask ensures the action prediction token $a_{in}$ at time $i$ cannot attend to the ground truth action at time $i$. Note that no other tokens attend to $a_{in}$ to stabilize the training process.
• Figure 3: (\ref{['fig:no-augmentation']}) Gap induced by video compression without data augmentation. We measured this gap by comparing model outputs on raw frames (inference) and resized frames (training). An irreducible gap arises from lossy video compression during data collection. The gap was smaller with RGB than YUV encoding and increases as compression quality degrades (lower file size). The x-axis qp denotes the quantization parameter, where larger values indicate lower quality. Data augmentation (\ref{['fig:augmentation']}) mitigates this gap when reasonable compression quality is used.
• Figure 4: (\ref{['fig:model-comparison']}) Human preference comparisons across model sizes (ties excluded). Each model is evaluated from nine game checkpoints, and its gameplay trajectories are recorded. Human evaluators then compare model pairs and judge which performs better based on (i) how closely the behavior resembles that of a human player and (ii) how effectively the model progresses toward the next checkpoint, when applicable. (\ref{['fig:instruction-following']}) Instruction-following comparison. Each model is evaluated from the same maze checkpoint with and without an instruction ("press the red button"), with five runs per condition. We report the success rate of completing the maze.
• Figure 5: Lowest test loss versus dataset size for the 1.2B model. As might be expected, we find the test loss fits a power-law curve closely.