Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StarBench: A Turn-Based RPG Benchmark for Agentic Multimodal Decision-Making and Information Seeking

Haoran Zhang, Chenhao Zhu, Sicong Guo, Hanzhe Guo, Haiming Li, Donglin Yu

TL;DR

StarBench tackles the problem of evaluating vision-language agents in a real-world, interactive RPG client by separating pixel-to-action grounding from information-seeking behavior. It introduces two matched regimes, DC and TA, plus an ask-or-act diagnostic across eight tasks to quantify how and when models seek guidance. The results show a large performance gap in direct grounding, and substantial gains when tool-assisted grounding and OCR-enhanced UI signals are available, with further uplift from targeted asking. This benchmark provides a reproducible, end-to-end testbed for advancing agentic multimodal decision-making in realistic, dynamic environments and informs the role of lightweight tooling and information seeking in practical AI systems.

Abstract

Human players do more than press buttons: they ground what they see on screen into precise keyboard-mouse actions and, when stuck, they seek information before trying again. We ask whether current vision-language models (VLMs) can do the same. Despite encouraging results under simplified control or tool scaffolds, human-like play in a real client - mapping raw screenshots to temporally coherent low-level actions while deciding when to ask for guidance - remains an open challenge. We introduce StarBench, a turn-based RPG benchmark derived from Honkai: Star Rail that targets these two human-like competencies: multimodal decision-making from pixels to actions and agentic information seeking. StarBench standardizes evaluation across eight combat tasks and two regimes with shared tasks and metrics: (i) direct control, where agents receive only screenshots and must emit low-level primitives (click and keypress) with no semantic hints; and (ii) tool-assisted control, where higher-level intents can be mapped to primitives by detectors and OCR outputs provide optional textualized observations to ease UI grounding. To mirror human practice, StarBench also includes an ask-or-act diagnostic that measures whether and when agents choose to request brief guidance before proceeding, and how that choice affects subsequent performance. We report reference baselines for contemporary VLMs and a human reference. Results expose sizable gaps in perception-to-control fidelity in the direct regime, while showing that judicious information seeking correlates with improved success, establishing StarBench as a reproducible yardstick for agentic information seeking and multimodal decision-making in real-client play.

StarBench: A Turn-Based RPG Benchmark for Agentic Multimodal Decision-Making and Information Seeking

TL;DR

StarBench tackles the problem of evaluating vision-language agents in a real-world, interactive RPG client by separating pixel-to-action grounding from information-seeking behavior. It introduces two matched regimes, DC and TA, plus an ask-or-act diagnostic across eight tasks to quantify how and when models seek guidance. The results show a large performance gap in direct grounding, and substantial gains when tool-assisted grounding and OCR-enhanced UI signals are available, with further uplift from targeted asking. This benchmark provides a reproducible, end-to-end testbed for advancing agentic multimodal decision-making in realistic, dynamic environments and informs the role of lightweight tooling and information seeking in practical AI systems.

Abstract

Human players do more than press buttons: they ground what they see on screen into precise keyboard-mouse actions and, when stuck, they seek information before trying again. We ask whether current vision-language models (VLMs) can do the same. Despite encouraging results under simplified control or tool scaffolds, human-like play in a real client - mapping raw screenshots to temporally coherent low-level actions while deciding when to ask for guidance - remains an open challenge. We introduce StarBench, a turn-based RPG benchmark derived from Honkai: Star Rail that targets these two human-like competencies: multimodal decision-making from pixels to actions and agentic information seeking. StarBench standardizes evaluation across eight combat tasks and two regimes with shared tasks and metrics: (i) direct control, where agents receive only screenshots and must emit low-level primitives (click and keypress) with no semantic hints; and (ii) tool-assisted control, where higher-level intents can be mapped to primitives by detectors and OCR outputs provide optional textualized observations to ease UI grounding. To mirror human practice, StarBench also includes an ask-or-act diagnostic that measures whether and when agents choose to request brief guidance before proceeding, and how that choice affects subsequent performance. We report reference baselines for contemporary VLMs and a human reference. Results expose sizable gaps in perception-to-control fidelity in the direct regime, while showing that judicious information seeking correlates with improved success, establishing StarBench as a reproducible yardstick for agentic information seeking and multimodal decision-making in real-client play.
Paper Structure (29 sections, 4 figures, 6 tables)

This paper contains 29 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Models
  4. Games as Benchmarks
  5. VLMs as Game Agents
  6. Background
  7. Problem Formalization
  8. HSR Combat Mechanics
  9. StarBench
  10. Interaction Protocol
  11. Direct Control (DC)
  12. Tool-Assisted (TA)
  13. Ask--or--Act Diagnostic
  14. Task Suite
  15. Echo of War.
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Annotated HSR combat UI. Key affordances include: (1) the action order track governing future turns; (2) Ultimate indicators (glow = ready); (3) team skill points; (4) Basic Attack and (5) Skill buttons with availability reflecting (3); (6) damage and status text; (7) the auto-battle toggle. Top-center, the boss frame shows a white toughness bar above a red HP bar. In the example, the team has 0 skill points, disabling (5) while (4) remains usable; the glow at (2) indicates Seele’s Ultimate can be cast, potentially off-turn.
  • Figure 2: Interaction Protocol: (i) DC (direct control) and (ii) TA (tool-assisted). Inputs are resized to fit each VLM’s native resolution, and outputs are resized back to the original scale for evaluation.
  • Figure 3: Ask--or--Act diagnostic pipeline. At episode start (TA regime), the agent chooses Ask (query LightRAG once for a short hint, persisted across the episode) or Act (proceed without guidance). Both paths run the same battle loop with the same action interface $a_t=(c,m,t)$; they differ only in prompt context. A decision log records concise summaries (actions sequence for previous battle and battle result) to support learning-to-ask.
  • Figure 4: Corpus composition by site. Forums: HoYoLABhoyolab, Redditreddit; Wiki: Fandomfandom, Game8game8, HoYoWikihoyowiki; Official site: Miyoushemiyoushe. Total: 1.14 MB (1138.39 KB).