AVA: Attentive VLM Agent for Mastering StarCraft II

TL;DR

The paper tackles the gap between AI agents and human perception in StarCraft II by introducing AVA, a multimodal vision-language agent that uses RGB visuals and natural language observations within the AVACraft environment. The core contribution is an integrated architecture combining Multimodal Priority Inference, Retrieval-Augmented Generation, and dynamic role assignment to achieve coherent, human-aligned tactics across multiple agents. Empirical results show AVA, powered by foundation models such as Qwen-VL and GPT-4o, can perform complex micromanagement tasks with minimal explicit training and outperform traditional MARL baselines in human-aligned metrics. This work advances multimodal game AI by demonstrating how human cognition-inspired perception and reasoning can be embedded in real-time RTS decision making, with implications for broader human-AI collaboration in complex domains.

Abstract

We introduce Attentive VLM Agent (AVA), a multimodal StarCraft II agent that aligns artificial agent perception with the human gameplay experience. Traditional frameworks such as SMAC rely on abstract state representations that diverge significantly from human perception, limiting the ecological validity of agent behavior. Our agent addresses this limitation by incorporating RGB visual inputs and natural language observations that more closely simulate human cognitive processes during gameplay. The AVA architecture consists of three integrated components: (1) a vision-language model enhanced with specialized self-attention mechanisms for strategic unit targeting and battlefield assessment, (2) a retrieval-augmented generation system that leverages domain-specific StarCraft II knowledge to inform tactical decisions, and (3) a dynamic role-based task distribution system that enables coordinated multi-agent behavior. The experimental evaluation in our proposed AVACraft environment, which contains 21 multimodal StarCraft II scenarios, demonstrates that AVA powered by foundation models (specifically Qwen-VL and GPT-4o) can execute complex tactical maneuvers without explicit training, achieving comparable performance to traditional MARL methods that require substantial training iterations. This work establishes a foundation for developing human-aligned StarCraft II agents and advances the broader research agenda of multimodal game AI. Our implementation is available at https://github.com/camel-ai/VLM-Play-StarCraft2.

Figures (12)

• Figure 1: Observation Space of AVACraft environment.
• Figure 2: VLM-Agent Architecture integrates Multimodal Priority Inference mechanisms, a knowledge-enhanced decision system, and a dynamic role assignment framework.
• Figure 3: Dynamic Role Assignment framework showing the evaluation and optimization process.
• Figure 4: Original RGB observation of battlefield situation in the Colossi vs Zerglings scenario.
• Figure 5: Annotated unit positions with unit IDs and health status.