Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAGE: Semantic-Aware Gray-Box Game Regression Testing with Large Language Models

Jinyu Cai, Jialong Li, Nianyu Li, Zhenyu Mao, Mingyue Zhang, Kenji Tei

TL;DR

SAGE tackles regression testing in gray-box live-service games by combining LLM-guided RL to generate a diverse foundation, semantic multi-objective optimization to prune to a high-value test subset, and LLM-based update-log analysis to prioritize tests for version changes. The framework yields higher bug diversity and near-human effectiveness while substantially reducing execution costs across two representative games. Ablation studies confirm the complementary roles of optimization and update-aware prioritization in achieving efficient, scalable regression testing. This approach enables automated, adaptive regression verification in complex game environments without source-code access, with strong practical implications for production pipelines.

Abstract

The rapid iteration cycles of modern live-service games make regression testing indispensable for maintaining quality and stability. However, existing regression testing approaches face critical limitations, especially in common gray-box settings where full source code access is unavailable: they heavily rely on manual effort for test case construction, struggle to maintain growing suites plagued by redundancy, and lack efficient mechanisms for prioritizing relevant tests. These challenges result in excessive testing costs, limited automation, and insufficient bug detection. To address these issues, we propose SAGE, a semanticaware regression testing framework for gray-box game environments. SAGE systematically addresses the core challenges of test generation, maintenance, and selection. It employs LLM-guided reinforcement learning for efficient, goal-oriented exploration to automatically generate a diverse foundational test suite. Subsequently, it applies a semantic-based multi-objective optimization to refine this suite into a compact, high-value subset by balancing cost, coverage, and rarity. Finally, it leverages LLM-based semantic analysis of update logs to prioritize test cases most relevant to version changes, enabling efficient adaptation across iterations. We evaluate SAGE on two representative environments, Overcooked Plus and Minecraft, comparing against both automated baselines and human-recorded test cases. Across all environments, SAGE achieves superior bug detection with significantly lower execution cost, while demonstrating strong adaptability to version updates.

SAGE: Semantic-Aware Gray-Box Game Regression Testing with Large Language Models

TL;DR

SAGE tackles regression testing in gray-box live-service games by combining LLM-guided RL to generate a diverse foundation, semantic multi-objective optimization to prune to a high-value test subset, and LLM-based update-log analysis to prioritize tests for version changes. The framework yields higher bug diversity and near-human effectiveness while substantially reducing execution costs across two representative games. Ablation studies confirm the complementary roles of optimization and update-aware prioritization in achieving efficient, scalable regression testing. This approach enables automated, adaptive regression verification in complex game environments without source-code access, with strong practical implications for production pipelines.

Abstract

The rapid iteration cycles of modern live-service games make regression testing indispensable for maintaining quality and stability. However, existing regression testing approaches face critical limitations, especially in common gray-box settings where full source code access is unavailable: they heavily rely on manual effort for test case construction, struggle to maintain growing suites plagued by redundancy, and lack efficient mechanisms for prioritizing relevant tests. These challenges result in excessive testing costs, limited automation, and insufficient bug detection. To address these issues, we propose SAGE, a semanticaware regression testing framework for gray-box game environments. SAGE systematically addresses the core challenges of test generation, maintenance, and selection. It employs LLM-guided reinforcement learning for efficient, goal-oriented exploration to automatically generate a diverse foundational test suite. Subsequently, it applies a semantic-based multi-objective optimization to refine this suite into a compact, high-value subset by balancing cost, coverage, and rarity. Finally, it leverages LLM-based semantic analysis of update logs to prioritize test cases most relevant to version changes, enabling efficient adaptation across iterations. We evaluate SAGE on two representative environments, Overcooked Plus and Minecraft, comparing against both automated baselines and human-recorded test cases. Across all environments, SAGE achieves superior bug detection with significantly lower execution cost, while demonstrating strong adaptability to version updates.

Paper Structure

This paper contains 36 sections, 7 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional Regression Testing Approaches
  4. Agent-Based Automated Testing Approaches
  5. Large Language Models in Software Testing and Games
  6. Position of This Study
  7. Proposal
  8. Overview
  9. LLM-RL Guided Test Case Generation
  10. Seed Trajectory Generation with LLMs
  11. Policy Learning and Guided Exploration
  12. Graph Construction and Test Case Derivation
  13. Multi-Objective Test Case Optimization
  14. Objective Formulation and Metric Definition
  15. Modeling Behavioral Rarity
  16. ...and 21 more sections

Figures (3)

  • Figure 1: Overview of SAGE. The framework consists of four main stages: (1) LLM-RL guided test case generation, (2) multi-objective test case optimization, (3) update-aware test case selection, and (4) test case execution and maintenance. These stages together enable automated, scalable, and update-sensitive regression testing in complex game environments. For better visibility and easier understanding, the state information ($s$) contained in test cases is omitted in this figure.
  • Figure 2: Illustration of game environments used in evaluation.
  • Figure 3: Ablation study of SAGE in Overcooked Plus. The figure shows the results of V2$\rightarrow$V3.