Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChatPCG: Large Language Model-Driven Reward Design for Procedural Content Generation

In-Chang Baek, Tae-Hwa Park, Jin-Ha Noh, Cheong-Mok Bae, Kyung-Joong Kim

TL;DR

Reward design for game AI is traditionally expert-driven and labor-intensive. The paper introduces ChatPCG, an LLM-driven framework that generates modular reward functions and uses a self-alignment loop guided by game logs, integrated with DRL for multiplayer content generation in RaidEnv II. Key contributions include a two-step reward design pipeline (insight generation and code implementation) with chain-of-thought based refinement, demonstrations of improved controllability, diversity, and team-building scores, and applicability to cooperative gameplay. This approach reduces reliance on human experts and enables automated, multi-objective content generation in complex multiplayer scenarios.

Abstract

Driven by the rapid growth of machine learning, recent advances in game artificial intelligence (AI) have significantly impacted productivity across various gaming genres. Reward design plays a pivotal role in training game AI models, wherein researchers implement concepts of specific reward functions. However, despite the presence of AI, the reward design process predominantly remains in the domain of human experts, as it is heavily reliant on their creativity and engineering skills. Therefore, this paper proposes ChatPCG, a large language model (LLM)-driven reward design framework.It leverages human-level insights, coupled with game expertise, to generate rewards tailored to specific game features automatically. Moreover, ChatPCG is integrated with deep reinforcement learning, demonstrating its potential for multiplayer game content generation tasks. The results suggest that the proposed LLM exhibits the capability to comprehend game mechanics and content generation tasks, enabling tailored content generation for a specified game. This study not only highlights the potential for improving accessibility in content generation but also aims to streamline the game AI development process.

ChatPCG: Large Language Model-Driven Reward Design for Procedural Content Generation

TL;DR

Reward design for game AI is traditionally expert-driven and labor-intensive. The paper introduces ChatPCG, an LLM-driven framework that generates modular reward functions and uses a self-alignment loop guided by game logs, integrated with DRL for multiplayer content generation in RaidEnv II. Key contributions include a two-step reward design pipeline (insight generation and code implementation) with chain-of-thought based refinement, demonstrations of improved controllability, diversity, and team-building scores, and applicability to cooperative gameplay. This approach reduces reliance on human experts and enables automated, multi-objective content generation in complex multiplayer scenarios.

Abstract

Driven by the rapid growth of machine learning, recent advances in game artificial intelligence (AI) have significantly impacted productivity across various gaming genres. Reward design plays a pivotal role in training game AI models, wherein researchers implement concepts of specific reward functions. However, despite the presence of AI, the reward design process predominantly remains in the domain of human experts, as it is heavily reliant on their creativity and engineering skills. Therefore, this paper proposes ChatPCG, a large language model (LLM)-driven reward design framework.It leverages human-level insights, coupled with game expertise, to generate rewards tailored to specific game features automatically. Moreover, ChatPCG is integrated with deep reinforcement learning, demonstrating its potential for multiplayer game content generation tasks. The results suggest that the proposed LLM exhibits the capability to comprehend game mechanics and content generation tasks, enabling tailored content generation for a specified game. This study not only highlights the potential for improving accessibility in content generation but also aims to streamline the game AI development process.
Paper Structure (17 sections, 2 equations, 2 figures, 1 table)

This paper contains 17 sections, 2 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Reward Generation Task
  4. Preliminaries
  5. RaidEnv II: Heterogeneous Character Generation
  6. Proposed Method
  7. ChatPCG Framework
  8. Insight to Reward Generation
  9. Self-alignment Process
  10. Reward Generation for Cooperative Game
  11. Experiment
  12. Comparisons
  13. Experiment Setting
  14. Evaluation Metrics
  15. Experimental Result
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Snapshot of the RaidEnv II environment. The four agents on the bottom-side are players and top-side green agent is the boss.
  • Figure 2: Architecture of ChatPCG framework. “Message icons” indicate the use of LMs in the context. Refer to Section \ref{['sec:method']} for detailed description.