CombatVLA: An Efficient Vision-Language-Action Model for Combat Tasks in 3D Action Role-Playing Games

TL;DR

CombatVLA presents an efficient 3B vision-language-action model tailored for real-time combat tasks in 3D ARPGs. The approach centers on action-of-thought data from an action tracker, a three-stage progressive learning regime, and a truncated AoT inference strategy, enabling rapid and accurate action execution. Empirical results show CombatVLA outperforms existing VLAs in combat understanding, while delivering up to 50x speedups in real-time gameplay and even surpassing human task success on certain tests. The work provides open-source resources, including the action tracker, AoT dataset, benchmark, model weights, and code, to accelerate future research and practical deployment.

Abstract

Recent advances in Vision-Language-Action models (VLAs) have expanded the capabilities of embodied intelligence. However, significant challenges remain in real-time decision-making in complex 3D environments, which demand second-level responses, high-resolution perception, and tactical reasoning under dynamic conditions. To advance the field, we introduce CombatVLA, an efficient VLA model optimized for combat tasks in 3D action role-playing games(ARPGs). Specifically, our CombatVLA is a 3B model trained on video-action pairs collected by an action tracker, where the data is formatted as action-of-thought (AoT) sequences. Thereafter, CombatVLA seamlessly integrates into an action execution framework, allowing efficient inference through our truncated AoT strategy. Experimental results demonstrate that CombatVLA not only outperforms all existing models on the combat understanding benchmark but also achieves a 50-fold acceleration in game combat. Moreover, it has a higher task success rate than human players. We will open-source all resources, including the action tracker, dataset, benchmark, model weights, training code, and the implementation of the framework at https://combatvla.github.io/.

Figures (10)

• Figure 1: CombatVLA surpasses GPT-4o and Qwen2.5-VL in combat understanding, is 50 times faster than Cradle and VARP framework, and has a higher success rate than humans.
• Figure 2: (a) An action tracker collects human data on keyboard and mouse use. (b) Three types of AoT training data collected by the action tracker are used for progressive learning. (c) Combat understanding benchmark (namely CUBench) assesses the model's combat IQ in three areas: gathering, comprehension, and reasoning. (d) CombatVLA model is trained on AoT data with the constraint of action alignment loss and modality contrastive loss. (e) Deployment of CombatVLA to operate real PCs.
• Figure 3: Combat understanding benchmark (i.e. CUBench) has three categories: gathering (single-image judgment), comprehension (multi-image judgment), and reasoning (multiple-image multiple-choice).
• Figure 4: Visualization of action-of-thought reasoning. Given a video or frame input, the model can quickly infer the correct action under the semantic guidance of AoT. The special token $\langle \text{TRUNC} \rangle$ speeds up efficient reasoning by truncating output.
• Figure 5: Comparison of task-level practical tests. Our CombatVLA not only outperforms all VLM-based agents (i.e., Cradle and VARP) but also has a higher task success rate than human players.