Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Combining Reinforcement Learning and Behavior Trees for NPCs in Video Games with AMD Schola

Tian Liu, Alex Cann, Ian Colbert, Mehdi Saeedi

TL;DR

The paper addresses the gap between reinforcement learning and traditional behavior trees for commercial game NPCs by proposing a BT+RL hybrid that combines RL-driven skill policies with a BT-driven control framework. Implemented in the AMD Schola plugin for Unreal Engine, it builds multi-skill NPCs inspired by The Last of Us and provides open-source environments and implementations. Through skill-specific PPO-based RL models, a BT baseline, and a curriculum RL baseline, the study evaluates in a controlled Unreal Engine shooter environment, demonstrating that the BT+RL approach achieves strong performance and better consistency than curriculum learning while approaching the efficiency and predictability of pure BTs. The work suggests that modular, reusable components and developer-controllable RL actions can make advanced NPCs more practical for commercial games, with future work focusing on optimization and broader task coverage.

Abstract

While the rapid advancements in the reinforcement learning (RL) research community have been remarkable, the adoption in commercial video games remains slow. In this paper, we outline common challenges the Game AI community faces when using RL-driven NPCs in practice, and highlight the intersection of RL with traditional behavior trees (BTs) as a crucial juncture to be explored further. Although the BT+RL intersection has been suggested in several research papers, its adoption is rare. We demonstrate the viability of this approach using AMD Schola -- a plugin for training RL agents in Unreal Engine -- by creating multi-task NPCs in a complex 3D environment inspired by the commercial video game ``The Last of Us". We provide detailed methodologies for jointly training RL models with BTs while showcasing various skills.

Combining Reinforcement Learning and Behavior Trees for NPCs in Video Games with AMD Schola

TL;DR

The paper addresses the gap between reinforcement learning and traditional behavior trees for commercial game NPCs by proposing a BT+RL hybrid that combines RL-driven skill policies with a BT-driven control framework. Implemented in the AMD Schola plugin for Unreal Engine, it builds multi-skill NPCs inspired by The Last of Us and provides open-source environments and implementations. Through skill-specific PPO-based RL models, a BT baseline, and a curriculum RL baseline, the study evaluates in a controlled Unreal Engine shooter environment, demonstrating that the BT+RL approach achieves strong performance and better consistency than curriculum learning while approaching the efficiency and predictability of pure BTs. The work suggests that modular, reusable components and developer-controllable RL actions can make advanced NPCs more practical for commercial games, with future work focusing on optimization and broader task coverage.

Abstract

While the rapid advancements in the reinforcement learning (RL) research community have been remarkable, the adoption in commercial video games remains slow. In this paper, we outline common challenges the Game AI community faces when using RL-driven NPCs in practice, and highlight the intersection of RL with traditional behavior trees (BTs) as a crucial juncture to be explored further. Although the BT+RL intersection has been suggested in several research papers, its adoption is rare. We demonstrate the viability of this approach using AMD Schola -- a plugin for training RL agents in Unreal Engine -- by creating multi-task NPCs in a complex 3D environment inspired by the commercial video game ``The Last of Us". We provide detailed methodologies for jointly training RL models with BTs while showcasing various skills.
Paper Structure (12 sections, 4 figures, 5 tables)

This paper contains 12 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Multi-Skill BT+RL NPCs
  3. RL-based Models
  4. Training Configuration
  5. Empirical Evaluation
  6. The Environment
  7. BT Baseline Model
  8. Curriculum Learning Baseline
  9. Results
  10. Model Quality
  11. Test time FPS
  12. Conclusion & Future Work

Figures (4)

  • Figure 1: We show the strategic decisions for different skills. Blue receptacles represent skills controlled by RL-based models. "$?$" and "-$>$" are Selector and Sequence nodes, respectively.
  • Figure 2: Demonstration of the NPC focusing on the Combat Skill.
  • Figure 3: Visualization of the learning curriculum for Curriculum RL agent.
  • Figure 4: The distribution of episode durations for the wins and losses of each method against the Aggressive NPC.