Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Human-AI Collaborative Game Testing with Vision Language Models

Boran Zhang, Muhan Xu, Zhijun Pan

TL;DR

The paper addresses the challenge of scalable quality assurance in increasingly complex video games by introducing an AI-assisted testing workflow that combines state-of-the-art multimodal AI with human oversight. Using GPT-4o to analyze screenshots and detect predefined defect categories, the study compares manual testing to AI-assisted methods across four conditions involving the presence or absence of AI support and knowledge resources, over 800 test cases with 276 participants. Key findings show that AI assistance substantially improves defect detection, especially when paired with detailed knowledge, though AI errors and hallucinations can detrimentally affect human judgment unless mitigated by context and critical evaluation training. Practically, the work offers actionable guidelines for integrating AI into game testing workflows, balancing automation with human review to maximize efficiency and accuracy while managing AI-related risks.

Abstract

As modern video games become increasingly complex, traditional manual testing methods are proving costly and inefficient, limiting the ability to ensure high-quality game experiences. While advancements in Artificial Intelligence (AI) offer the potential to assist human testers, the effectiveness of AI in truly enhancing real-world human performance remains underexplored. This study investigates how AI can improve game testing by developing and experimenting with an AI-assisted workflow that leverages state-of-the-art machine learning models for defect detection. Through an experiment involving 800 test cases and 276 participants of varying backgrounds, we evaluate the effectiveness of AI assistance under four conditions: with or without AI support, and with or without detailed knowledge of defects and design documentation. The results indicate that AI assistance significantly improves defect identification performance, particularly when paired with detailed knowledge. However, challenges arise when AI errors occur, negatively impacting human decision-making. Our findings show the importance of optimizing human-AI collaboration and implementing strategies to mitigate the effects of AI inaccuracies. By this research, we demonstrate AI's potential and problems in enhancing efficiency and accuracy in game testing workflows and offers practical insights for integrating AI into the testing process.

Human-AI Collaborative Game Testing with Vision Language Models

TL;DR

The paper addresses the challenge of scalable quality assurance in increasingly complex video games by introducing an AI-assisted testing workflow that combines state-of-the-art multimodal AI with human oversight. Using GPT-4o to analyze screenshots and detect predefined defect categories, the study compares manual testing to AI-assisted methods across four conditions involving the presence or absence of AI support and knowledge resources, over 800 test cases with 276 participants. Key findings show that AI assistance substantially improves defect detection, especially when paired with detailed knowledge, though AI errors and hallucinations can detrimentally affect human judgment unless mitigated by context and critical evaluation training. Practically, the work offers actionable guidelines for integrating AI into game testing workflows, balancing automation with human review to maximize efficiency and accuracy while managing AI-related risks.

Abstract

As modern video games become increasingly complex, traditional manual testing methods are proving costly and inefficient, limiting the ability to ensure high-quality game experiences. While advancements in Artificial Intelligence (AI) offer the potential to assist human testers, the effectiveness of AI in truly enhancing real-world human performance remains underexplored. This study investigates how AI can improve game testing by developing and experimenting with an AI-assisted workflow that leverages state-of-the-art machine learning models for defect detection. Through an experiment involving 800 test cases and 276 participants of varying backgrounds, we evaluate the effectiveness of AI assistance under four conditions: with or without AI support, and with or without detailed knowledge of defects and design documentation. The results indicate that AI assistance significantly improves defect identification performance, particularly when paired with detailed knowledge. However, challenges arise when AI errors occur, negatively impacting human decision-making. Our findings show the importance of optimizing human-AI collaboration and implementing strategies to mitigate the effects of AI inaccuracies. By this research, we demonstrate AI's potential and problems in enhancing efficiency and accuracy in game testing workflows and offers practical insights for integrating AI into the testing process.
Paper Structure (47 sections, 4 equations, 3 figures, 5 tables)

This paper contains 47 sections, 4 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal AI
  4. Human-AI Collaboration
  5. AI-Assisted Game Testing
  6. Methodology
  7. Game Testing Workflow
  8. Experiment Setup
  9. System
  10. Test Cases and Execution
  11. Scoring Mechanism
  12. Participants
  13. Human Participants
  14. AI Component
  15. Tasks
  16. ...and 32 more sections

Figures (3)

  • Figure 1: Traditional Game Testing Workflow (Simplified)
  • Figure 2: AI-Assisted Game Testing Workflow (Simplified)
  • Figure 3: Test accuracy distribution. This image illustrates the maximum, average, upper quartile, median, lower quartile, and minimum accuracies of participants under different testing conditions.