Learning to play: A Multimodal Agent for 3D Game-Play

TL;DR

This work tackles real-time, multimodal control in 3-D first-person games by building a large, text-annotated dataset and a text-conditioned policy trained via behavior cloning. A multi-game inverse dynamics model (IDM) imputes actions on unlabeled videos, enabling extensive pretraining for a decoder-only Transformer policy named Pixels2Play (P2P0.3) that runs in real time on consumer GPUs. The approach demonstrates cross-game generalization, enabling text-conditioned play across Roblox, MS-DOS titles, and FPS/racing games, with qualitative and limited quantitative evidence, while identifying challenges in long-horizon planning and robust evaluation. The work highlights the practical impact of leveraging unlabeled video through IDM, compact token-based architectures, and text conditioning to advance multimodal gaming agents on accessible hardware.

Abstract

We argue that 3-D first-person video games are a challenging environment for real-time multi-modal reasoning. We first describe our dataset of human game-play, collected across a large variety of 3-D first-person games, which is both substantially larger and more diverse compared to prior publicly disclosed datasets, and contains text instructions. We demonstrate that we can learn an inverse dynamics model from this dataset, which allows us to impute actions on a much larger dataset of publicly available videos of human game play that lack recorded actions. We then train a text-conditioned agent for game playing using behavior cloning, with a custom architecture capable of realtime inference on a consumer GPU. We show the resulting model is capable of playing a variety of 3-D games and responding to text input. Finally, we outline some of the remaining challenges such as long-horizon tasks and quantitative evaluation across a large set of games.

Figures (13)

• Figure 1: \ref{['subfig:model']} Architecture of P2P0.3. The core policy transformer and action decoder are both decoder-only transformers. Each timestep begins with a text token $x_i$. Since many frame do not contain a text annotation there is a learned default value $x_{null}$ inputed on these frames. This is followed by $\leq4$ image tokens from video frame $o_{i}$ followed by a learnable “reasoning’’ token $t_{i}$ that grants the model extra computation time. The policy transformer then outputs a single action token $\tilde{a}_i$. A smaller transformer, the action decoder, then auto-regressively transforms and samples the single action token into the full action space. \ref{['subfig:mask']} Attention mask used in our transformer policy (yellow denotes $1$ and blue $0$). Tick marks show the boundaries of successive inputs. The mask can be read by looking on the x-axis for the query and translating up to see the parts of the key masked out, e.g. tokens in $o_1$ can attend to $o_1$ and $t_1$ but not the future. Observe in step 2 that the prior action token $a_1$ is masked out.
• Figure 2: Validation metrics for finetuned models compared with baseline model trained only on labeled data. We consistently observe fine-tuning the pre-trained model results in improved performance, indicating the additional unlabelled data improves model generalization.
• Figure 3: Examples of a MS-DOS game, Roblox, and first person shooters P2P0.3 is currently capable of playing. For simple games such as MS-DOS Need For Speed or Roblox "Be a Snake" or simple FPS the model is capable of gameplay at the level of non-expert human. However, for games which require longer-term planning such the model struggles to complete levels. Video of gameplay can be seen view at https://blog.player2.game/p/pixels2play-v03-text-conditioned.
• Figure 4: Validation perplexity
• Figure 5: To evaluate the text conditioning we start the model from the same checkpoint with differing text prompts.