DynamicVLA: A Vision-Language-Action Model for Dynamic Object Manipulation

TL;DR

DynamicVLA tackles dynamic object manipulation by bridging perception and action under moving targets with a compact 0.4B VLA, Continuous Inference, and Latent-aware Action Streaming. It introduces DOM, a scalable benchmark with automated simulation and real-world data across multiple embodiments (200K synthetic, 2K real-world episodes across 2.8K scenes and 206 objects). The approach yields faster, more reliable perception–action coupling and improved generalization to unseen objects and scenes, demonstrated across nine evaluation dimensions and real-world trials. Together, these contributions provide a practical path toward general dynamic manipulation and scalable data infrastructure for future research. The findings highlight temporal misalignment as a key bottleneck and propose latency-aware architectures as a general design principle.

Abstract

Manipulating dynamic objects remains an open challenge for Vision-Language-Action (VLA) models, which, despite strong generalization in static manipulation, struggle in dynamic scenarios requiring rapid perception, temporal anticipation, and continuous control. We present DynamicVLA, a framework for dynamic object manipulation that integrates temporal reasoning and closed-loop adaptation through three key designs: 1) a compact 0.4B VLA using a convolutional vision encoder for spatially efficient, structurally faithful encoding, enabling fast multimodal inference; 2) Continuous Inference, enabling overlapping reasoning and execution for lower latency and timely adaptation to object motion; and 3) Latent-aware Action Streaming, which bridges the perception-execution gap by enforcing temporally aligned action execution. To fill the missing foundation of dynamic manipulation data, we introduce the Dynamic Object Manipulation (DOM) benchmark, built from scratch with an auto data collection pipeline that efficiently gathers 200K synthetic episodes across 2.8K scenes and 206 objects, and enables fast collection of 2K real-world episodes without teleoperation. Extensive evaluations demonstrate remarkable improvements in response speed, perception, and generalization, positioning DynamicVLA as a unified framework for general dynamic object manipulation across embodiments.

Figures (6)

• Figure 1: (a) Current VLA models face perception–execution (P.E.) gaps and inter-chunk waiting, causing delayed reactions to dynamic objects. (b) DynamicVLA addresses these issues through Latent-Aware Action Streaming (LAAS) and Continuous Inference, eliminating both gaps and waiting for seamless action transitions. (c) The Dynamic Object Manipulation (DOM) benchmark, built from scratch, features 2.8K scenes and 206 objects for evaluating Perception, Interaction, and Generalization, while its auto data collection pipeline enables efficient gathering of 200K synthetic and 2K real-world episodes.
• Figure 2: Overview of DynamicVLA.(a) A 0.4B-parameter VLA architecture couples a lightweight backbone with an action expert for fast closed-loop control. (b) Continuous Inference overlaps inference and execution through pipelined inference windows, enabling non-blocking action execution across consecutive action chunks. (c) Latent-aware Action Streaming enforces temporally consistent execution by invalidating outdated actions and prioritizing actions from the most recent action chunk.
• Figure 3: Automatic Simulation and Real-world Data Collection.Environment Setup: simulation and real-world settings share diverse objects, tabletop scenes, and synchronized multiview cameras. Object State Acquisition: simulation provides ground-truth 6D object states, while real-world multiview RGB observations are converted into a real-world "simulator" interface that enables automatic dynamic-manipulation data collection without teleoperation or ground-truth sensing. State-machine Controller: a shared four-stage controller uses these states to execute approach, grasp, place, and reset behaviors.
• Figure 4: Real-world Interaction Evaluation. We compare representative VLA models on six real-world dynamic manipulation tasks across Franka and PiPER, averaging success rates over 20 trials for each of three paired motion–position configurations, with object motion generated by a secondary robot arm.
• Figure 5: Real-world Perception Evaluation. We compare representative VLA models on six real-world dynamic manipulation tasks across Franka and PiPER, averaging success rates over 20 trials for each of three paired motion–position configurations, with object motion generated by a secondary robot arm.