NitroGen: An Open Foundation Model for Generalist Gaming Agents

TL;DR

NitroGen addresses the challenge of generalist embodied agents by scaling data and learning to operate across many games without handcrafted programmatic interfaces. It combines an internet-scale dataset of action-labeled gameplay, a universal simulator wrapper, and a vision–action transformer trained with conditional flow matching to produce zero-shot gameplay across 1,000+ games. The work demonstrates strong cross-game generalization, with notable transfer gains (up to 52% relative improvement in low-data regimes) when fine-tuning on unseen games, and provides open-source dataset, simulator, and model weights to accelerate research in generalist embodied AI. This approach offers a scalable path toward generalist agents capable of adapting to diverse, unseen game environments without extensive per-title tailoring.

Abstract

We introduce NitroGen, a vision-action foundation model for generalist gaming agents that is trained on 40,000 hours of gameplay videos across more than 1,000 games. We incorporate three key ingredients: 1) an internet-scale video-action dataset constructed by automatically extracting player actions from publicly available gameplay videos, 2) a multi-game benchmark environment that can measure cross-game generalization, and 3) a unified vision-action model trained with large-scale behavior cloning. NitroGen exhibits strong competence across diverse domains, including combat encounters in 3D action games, high-precision control in 2D platformers, and exploration in procedurally generated worlds. It transfers effectively to unseen games, achieving up to 52% relative improvement in task success rates over models trained from scratch. We release the dataset, evaluation suite, and model weights to advance research on generalist embodied agents.

Figures (7)

• Figure 1: NitroGen overview.NitroGen consists of three main components: (1) Multi-game foundation agent (center) - a generalist vision-action model that takes in game observations and generates gamepad actions, enabling zero-shot gameplay across multiple titles and serving as a foundation for fine-tuning on new games; (2) Universal simulator (left) - an environment wrapper that allows any commercial game to be controlled through a Gymnasium API; and (3) Internet-scale dataset (right) - the largest and most diverse open-source gaming dataset curated from 40,000 hours of publicly available gaming videos, spanning more than 1,000 games with extracted action labels.
• Figure 2: Video-action dataset pipeline overview. We extract actions from on-screen displays which show the gamepad actions of the player in real-time; called "input overlays". (a) Dataset curation. We collect publicly available videos displaying a “gamepad overlay”. The diversity of these overlays presents significant challenges, as gamepads vary widely across content creators in controller types (e.g., Xbox, PlayStation, or others), transparency levels, and visual artifacts introduced by video compression. (b) Action extraction. For each collected video, we localize the gamepad by sampling 25 frames and running keypoint matching against a curated set of templates using SIFT and XFeat features. We use the template-matching results to localize and crop the gamepad region from each video. A hybrid classification–segmentation network is then trained to predict joystick positions and button states from the cropped controller images, enabling accurate reconstruction of player inputs.
• Figure 3: Distribution of the NitroGen dataset across games and genres. After filtering, the NitroGen dataset contains 40,000 hours of gameplay videos spanning more than 1,000 games. (a) Hours per game shows broad coverage, with 846 games having over one hour of data, 91 games with over 100 hours, and 15 games exceeding 1,000 hours each. (b) Genre distribution reveals Action-RPG games are most common (34.9% of total hours), followed by Platformer (18.4%) and Action-Adventure (9.2%) games, with the remainder distributed across seven genres.
• Figure 4: In-game rollouts. We show NitroGen performing tasks in diverse 2D and 3D environments. These tasks can take from a few seconds to a few minutes to perform. Some of them include memorization, while others are performed in procedurally generated worlds and require the model to adapt.
• Figure 5: Gamepad parsing performance for different controller families. We verify the correctness of our action extraction pipeline by comparing performance across different controller families against ground-truth data. (a) shows joystick R² correlation scores (averaged for both left and right joysticks) with an overall average of 0.84. (b) shows button frame accuracy with an overall average of 0.96.