Selective Perception for Robot: Task-Aware Attention in Multimodal VLA

TL;DR

This work tackles the inefficiency and noise of static multimodal fusion in Vision-Language-Action robotics by introducing a context-aware Camera Router that dynamically gates information from multiple sensory streams. The router uses wrist-camera input and a task prompt to predict modality relevance, enabling task-driven attention and reduced computation via a gating mechanism. Training combines a Flow-Matching VLA backbone with LoRA fine-tuning, supervised by a routing loss and augmented by a VLM-based auto-labeling pipeline to scale supervision without human labeling. Experiments across three industrial tasks demonstrate substantial gains in success rates and inference efficiency, with VLM-derived labels approaching human labeling quality and offering a viable path to scalable robot learning. The approach promises improved robustness for long-horizon manipulation under resource constraints and lays groundwork for incorporating additional feedback modalities such as force data in future work.

Abstract

In robotics, Vision-Language-Action (VLA) models that integrate diverse multimodal signals from multi-view inputs have emerged as an effective approach. However, most prior work adopts static fusion that processes all visual inputs uniformly, which incurs unnecessary computational overhead and allows task-irrelevant background information to act as noise. Inspired by the principles of human active perception, we propose a dynamic information fusion framework designed to maximize the efficiency and robustness of VLA models. Our approach introduces a lightweight adaptive routing architecture that analyzes the current text prompt and observations from a wrist-mounted camera in real-time to predict the task-relevance of multiple camera views. By conditionally attenuating computations for views with low informational utility and selectively providing only essential visual features to the policy network, Our framework achieves computation efficiency proportional to task relevance. Furthermore, to efficiently secure large-scale annotation data for router training, we established an automated labeling pipeline utilizing Vision-Language Models (VLMs) to minimize data collection and annotation costs. Experimental results in real-world robotic manipulation scenarios demonstrate that the proposed approach achieves significant improvements in both inference efficiency and control performance compared to existing VLA models, validating the effectiveness and practicality of dynamic information fusion in resource-constrained, real-time robot control environments.

Figures (8)

• Figure 1: Operational process of the proposed model. (a) Camera Router Module: The router utilizes the wrist camera and text prompt to analyze the current context and predict the real-time importance of each modality. (b) Task-Aware Attention: Based on the router's prediction, the model selectively attends to the most informative visual streams. (c) Real-world Execution: Robot action sequence for Task 3 (Rotate the valve on the hot side and hold down the front button).
• Figure 2: Overview of the proposed framework. (a) Experiment Setup (b) Architecture of the proposed framework and the dynamic router. The router takes the wrist camera and langage prompt feed as inputs to predict importance weights for multiple visual sources, including external, thermal, and tactile cameras, thereby prioritizing task-relevant sensor information.
• Figure 3: Overview of VLA Training with Dynamic Router. The Camera Router predicts importance weights ($W$) for multiple camera views, acting as a gating mechanism. This allows the VLA framework to selectively prioritize task-relevant visual information during the action training process.
• Figure 4: Experimental Setup for Each Task. (a) Battery Sorting, (b) Wire Selection, and (c) Valve Operation. While all tasks share a dual RGB camera system for visual perception, each setup incorporates task-specific modalities—such as thermal cameras or tactile sensors—to provide complementary sensory information.
• Figure 5: Performance on Long-Horizon Sequential Tasks. (a) Illustration of the three-stage task progression (b) Success rates for each sub-task. Our method demonstrates sustained performance across the entire sequence, highlighting its robustness in long-horizon manipulation scenarios.