MAIN-VLA: Modeling Abstraction of Intention and eNvironment for Vision-Language-Action Models

TL;DR

MAIN-VLA tackles perceptual overload in open-world Vision-Language-Action tasks by explicitly modeling Intention Abstraction and Environment Semantics Abstraction, grounding decisions in compact semantic primitives and task-relevant affordances. A unified causal Transformer acts as a conscious bottleneck, with IA compressing verbal instructions into actionable primitives and ESA projecting visuals into a sparse spatial grid, enabling a parameter-free token pruning mechanism. Across Minecraft and large-scale PvP benchmarks, MAIN-VLA achieves state-of-the-art decision quality, stronger generalization, and substantial inference efficiency, while ablations confirm the complementary value of IA, ESA, and pruning. The results suggest that explicit cross-modal semantic abstraction with emergent sparsity can robustly empower embodied agents in dynamic, noisy environments.

Abstract

Despite significant progress in Visual-Language-Action (VLA), in highly complex and dynamic environments that involve real-time unpredictable interactions (such as 3D open worlds and large-scale PvP games), existing approaches remain inefficient at extracting action-critical signals from redundant sensor streams. To tackle this, we introduce MAIN-VLA, a framework that explicitly Models the Abstraction of Intention and eNvironment to ground decision-making in deep semantic alignment rather than superficial pattern matching. Specifically, our Intention Abstraction (IA) extracts verbose linguistic instructions and their associated reasoning into compact, explicit semantic primitives, while the Environment Semantics Abstraction (ESA) projects overwhelming visual streams into a structured, topological affordance representation. Furthermore, aligning these two abstract modalities induces an emergent attention-concentration effect, enabling a parameter-free token-pruning strategy that filters out perceptual redundancy without degrading performance. Extensive experiments in open-world Minecraft and large-scale PvP environments (Game for Peace and Valorant) demonstrate that MAIN-VLA sets a new state-of-the-art, which achieves superior decision quality, stronger generalization, and cutting-edge inference efficiency.

Figures (4)

• Figure 1: Unlike conventional VLA models that map low-level inputs directly to actions, our MAIN-VLA explicitly constructing Intention Abstraction (IA) and Environment Semantics Abstraction (ESA) through multi-modal de-tokenizers. As illustrated above, within a PvP combat scenario such as Game for Peace, our MAIN-VLA grounds multimodal inputs (task instructions and visual scenes) into high-level, interpretable concepts, e.g., "wall" and "enemy". When trained on action sequences like "move left (to use the wall for cover), then aim and shoot," the model grounds the semantic relationship between "wall" and "move left," recognizing not just the object's presence but also its functional role as tactical cover. This two-tier abstraction enables the model to reason over spatial and functional relationships in dynamic environments.
• Figure 2: Overview Framework. During training, the Intention Abstraction (IA) and Environment Semantics Abstraction (ESA) pathways align instructions and visual inputs into sparse, actionable primitives. At inference, MAIN-VLA prunes perceptual redundancies by retaining only top-$K$ pruned task-critical tokens. This overall pipeline mimics the human conscious bottleneck by integrating semantic abstraction with dynamic pruning, explicitly filtering sensory overload to achieve efficient, low-latency embodied behavior.
• Figure 3: The data constrain pipeline of Intention Abstraction (IA). A foundation model queries a domain-specific knowledge base via retrieval-augmented generation (RAG) to synthesize a detailed full intention and reasoning description of the video trajectory. This detailed reasoning is extracted into an ordered sequence of discrete keywords to provide supervision for the hindsight intention alignment objective.
• Figure 4: Resilience to aggressive token pruning. While the JARVIS-VLA (gray) collapses under information loss, MAIN-VLA (red) maintains near-invariant performance even at 75% pruning. This enables a $4 \times$ inference acceleration (blue bars) with negligible performance drop, confirming our abstractions effectively filter redundant visual noise.