Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agents Play Thousands of 3D Video Games

Zhongwen Xu, Xianliang Wang, Siyi Li, Tao Yu, Liang Wang, Qiang Fu, Wei Yang

TL;DR

PORTAL reframes game AI as language-guided policy generation, using LLMs to design domain-specific language representations of behavior trees that are executed by lightweight neural and rule-based components. By decoupling strategic planning from low-level control and employing a reflexion loop with quantitative metrics plus vision-language analysis, PORTAL achieves rapid, cross-game generalization across thousands of 3D FPS environments while maintaining interpretability. The approach relies on offline policy generation with iterative refinement, a meta-coordination network to select among pre-generated policies, and an instant development pipeline that exports executable policies as JSON for in-game deployment. These innovations reduce the computational burden of traditional RL, enable real-time behavior adaptation, and provide a practical path toward scalable game AI across varied genres. The empirical results on FPS games demonstrate improved development efficiency, policy generalization, and behavioral diversity, signaling substantial practical impact for commercial games and beyond.

Abstract

We present PORTAL, a novel framework for developing artificial intelligence agents capable of playing thousands of 3D video games through language-guided policy generation. By transforming decision-making problems into language modeling tasks, our approach leverages large language models (LLMs) to generate behavior trees represented in domain-specific language (DSL). This method eliminates the computational burden associated with traditional reinforcement learning approaches while preserving strategic depth and rapid adaptability. Our framework introduces a hybrid policy structure that combines rule-based nodes with neural network components, enabling both high-level strategic reasoning and precise low-level control. A dual-feedback mechanism incorporating quantitative game metrics and vision-language model analysis facilitates iterative policy improvement at both tactical and strategic levels. The resulting policies are instantaneously deployable, human-interpretable, and capable of generalizing across diverse gaming environments. Experimental results demonstrate PORTAL's effectiveness across thousands of first-person shooter (FPS) games, showcasing significant improvements in development efficiency, policy generalization, and behavior diversity compared to traditional approaches. PORTAL represents a significant advancement in game AI development, offering a practical solution for creating sophisticated agents that can operate across thousands of commercial video games with minimal development overhead. Experiment results on the 3D video games are best viewed on https://zhongwen.one/projects/portal .

Agents Play Thousands of 3D Video Games

TL;DR

PORTAL reframes game AI as language-guided policy generation, using LLMs to design domain-specific language representations of behavior trees that are executed by lightweight neural and rule-based components. By decoupling strategic planning from low-level control and employing a reflexion loop with quantitative metrics plus vision-language analysis, PORTAL achieves rapid, cross-game generalization across thousands of 3D FPS environments while maintaining interpretability. The approach relies on offline policy generation with iterative refinement, a meta-coordination network to select among pre-generated policies, and an instant development pipeline that exports executable policies as JSON for in-game deployment. These innovations reduce the computational burden of traditional RL, enable real-time behavior adaptation, and provide a practical path toward scalable game AI across varied genres. The empirical results on FPS games demonstrate improved development efficiency, policy generalization, and behavioral diversity, signaling substantial practical impact for commercial games and beyond.

Abstract

We present PORTAL, a novel framework for developing artificial intelligence agents capable of playing thousands of 3D video games through language-guided policy generation. By transforming decision-making problems into language modeling tasks, our approach leverages large language models (LLMs) to generate behavior trees represented in domain-specific language (DSL). This method eliminates the computational burden associated with traditional reinforcement learning approaches while preserving strategic depth and rapid adaptability. Our framework introduces a hybrid policy structure that combines rule-based nodes with neural network components, enabling both high-level strategic reasoning and precise low-level control. A dual-feedback mechanism incorporating quantitative game metrics and vision-language model analysis facilitates iterative policy improvement at both tactical and strategic levels. The resulting policies are instantaneously deployable, human-interpretable, and capable of generalizing across diverse gaming environments. Experimental results demonstrate PORTAL's effectiveness across thousands of first-person shooter (FPS) games, showcasing significant improvements in development efficiency, policy generalization, and behavior diversity compared to traditional approaches. PORTAL represents a significant advancement in game AI development, offering a practical solution for creating sophisticated agents that can operate across thousands of commercial video games with minimal development overhead. Experiment results on the 3D video games are best viewed on https://zhongwen.one/projects/portal .

Paper Structure

This paper contains 18 sections, 3 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Overview
  5. Domain-specific Language Representation
  6. Extensions Beyond Classic Behavior Trees
  7. An Agent-Environment Loop
  8. Chain-of-Thought for Tree Generation and Reflexion
  9. Sampling and Post-training
  10. A Policy Network to Schedule Policies
  11. Experiments
  12. Optimization for Game Metrics
  13. Vision-Language Models for Reflexion
  14. Playing Thousands of First-Person Shooter (FPS) Games
  15. Instant Development Procedure
  16. ...and 3 more sections

Figures (8)

  • Figure 1: Overview of the behavior tree generation process using LLMs.
  • Figure 2: System architecture showing the integration of LLM-based behavior tree generation with environment interaction and reflexion
  • Figure 3: The LLM-based behavior tree generation and refinement process
  • Figure 4: Two independent runs of Breadth-first Search (BFS).
  • Figure 5: Effects of VLM reflexion on game plays.
  • ...and 3 more figures

Theorems & Definitions (1)

  • Definition 3.1