Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case

Peng Chen, Pi Bu, Jun Song, Yuan Gao, Bo Zheng

TL;DR

A novel VARP (Vision Action Role-Playing) agent framework is proposed, consisting of an action planning system and a visual trajectory system that demonstrates the ability to perform basic tasks and succeed in 90% of easy and medium-level combat scenarios.

Abstract

Recently, large language model (LLM)-based agents have made significant advances across various fields. One of the most popular research areas involves applying these agents to video games. Traditionally, these methods have relied on game APIs to access in-game environmental and action data. However, this approach is limited by the availability of APIs and does not reflect how humans play games. With the advent of vision language models (VLMs), agents now have enhanced visual understanding capabilities, enabling them to interact with games using only visual inputs. Despite these advances, current approaches still face challenges in action-oriented tasks, particularly in action role-playing games (ARPGs), where reinforcement learning methods are prevalent but suffer from poor generalization and require extensive training. To address these limitations, we select an ARPG, ``Black Myth: Wukong'', as a research platform to explore the capability boundaries of existing VLMs in scenarios requiring visual-only input and complex action output. We define 12 tasks within the game, with 75% focusing on combat, and incorporate several state-of-the-art VLMs into this benchmark. Additionally, we will release a human operation dataset containing recorded gameplay videos and operation logs, including mouse and keyboard actions. Moreover, we propose a novel VARP (Vision Action Role-Playing) agent framework, consisting of an action planning system and a visual trajectory system. Our framework demonstrates the ability to perform basic tasks and succeed in 90% of easy and medium-level combat scenarios. This research aims to provide new insights and directions for applying multimodal agents in complex action game environments. The code and datasets will be made available at https://varp-agent.github.io/.

Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case

TL;DR

A novel VARP (Vision Action Role-Playing) agent framework is proposed, consisting of an action planning system and a visual trajectory system that demonstrates the ability to perform basic tasks and succeed in 90% of easy and medium-level combat scenarios.

Abstract

Recently, large language model (LLM)-based agents have made significant advances across various fields. One of the most popular research areas involves applying these agents to video games. Traditionally, these methods have relied on game APIs to access in-game environmental and action data. However, this approach is limited by the availability of APIs and does not reflect how humans play games. With the advent of vision language models (VLMs), agents now have enhanced visual understanding capabilities, enabling them to interact with games using only visual inputs. Despite these advances, current approaches still face challenges in action-oriented tasks, particularly in action role-playing games (ARPGs), where reinforcement learning methods are prevalent but suffer from poor generalization and require extensive training. To address these limitations, we select an ARPG, ``Black Myth: Wukong'', as a research platform to explore the capability boundaries of existing VLMs in scenarios requiring visual-only input and complex action output. We define 12 tasks within the game, with 75% focusing on combat, and incorporate several state-of-the-art VLMs into this benchmark. Additionally, we will release a human operation dataset containing recorded gameplay videos and operation logs, including mouse and keyboard actions. Moreover, we propose a novel VARP (Vision Action Role-Playing) agent framework, consisting of an action planning system and a visual trajectory system. Our framework demonstrates the ability to perform basic tasks and succeed in 90% of easy and medium-level combat scenarios. This research aims to provide new insights and directions for applying multimodal agents in complex action game environments. The code and datasets will be made available at https://varp-agent.github.io/.
Paper Structure (27 sections, 18 figures, 3 tables)

This paper contains 27 sections, 18 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overview
  4. Action Planning System
  5. Basic VLMs Group
  6. Self-Optimizable Action Generation Module
  7. Decomposable Task-Specific Auxiliary Modules
  8. Human-Guided Trajectory System
  9. Experiments
  10. Dataset Collection
  11. Benchmark and Task Definition
  12. Implementation Details
  13. Performance Evaluation
  14. Ablation Study
  15. Case Studies
  16. ...and 12 more sections

Figures (18)

  • Figure 1: Pipeline of VARP. We propose a novel framework named the VARP agent, which directly takes game screenshots as input and generates keyboard and mouse operations to play the ARPG.
  • Figure 2: Task 1
  • Figure 3: Task 2
  • Figure 4: Task 3
  • Figure 5: Task 4
  • ...and 13 more figures