Real-Time Robot Execution with Masked Action Chunking

TL;DR

The paper tackles real-time robotic manipulation under asynchronous VLA inference by identifying intra-chunk inconsistency and inter-chunk discontinuity as key failure modes. It introduces REMAC, a training-time approach with masked action chunking, prefix masking, a self-conditioned curriculum, and residual alignment, plus a prefix-preserved sampling scheme, all implemented with a lightweight LoRA-based adaptation that adds no inference delay. The method yields a delay-aware policy $\hat{\mathbf{v}}_{\pi}(\mathbf{A}_t|\mathbf{o}_t,d)$ that remains robust as the inference delay $d$ varies, achieving faster task execution and higher completion rates in both 12-task simulation (Kinetix) and real-world grasp-and-place experiments. REMAC also integrates with existing test-time corrections, enhancing backbone policies without incurring extra latency, which has practical implications for deploying real-time robotic systems across diverse hardware and network conditions.

Abstract

Real-time execution is essential for cyber-physical systems such as robots. These systems operate in dynamic real-world environments where even small delays can undermine responsiveness and compromise performance. Asynchronous inference has recently emerged as a system-level paradigm for real-time robot manipulation, enabling the next action chunk to be predicted while the current one is being executed. While this approach achieves real-time responsiveness, naive integration often results in execution failure. Previous methods attributed this failure to inter-chunk discontinuity and developed test-time algorithms to smooth chunk boundaries. In contrast, we identify another critical yet overlooked factor: intra-chunk inconsistency, where the robot's executed action chunk partially misaligns with its current perception. To address this, we propose REMAC, which learns corrective adjustments on the pretrained policy through masked action chunking, enabling the policy to remain resilient under mismatches between intended actions and actual execution during asynchronous inference. In addition, we introduce a prefix-preserved sampling procedure to reinforce inter-chunk continuity. Overall, our method delivers more reliable policies without incurring additional latency. Extensive experiments in both simulation and real-world settings demonstrate that our method enables faster task execution, maintains robustness across varying delays, and consistently achieves higher completion rates.

Figures (12)

• Figure 1: Illustration of execution paradigms. Arrowed lines of the same style indicate processes occurring simultaneously. (a) Synchronous inference: VLA prediction and robot execution alternate sequentially. (b) Asynchronous inference: VLA prediction runs concurrently with execution. (c) Although asynchronous inference enables real-time execution, it introduces two performance-degrading challenges: exacerbated inter-chunk discontinuity and intra-chunk inconsistency.
• Figure 1: Effectiveness of the different components in REMAC.
• Figure 2: Performance comparison in Kinetix environments. Left: Solve rates for individual tasks under varying inference delays. Right (top): Average performance across all environments. Our method consistently outperforms baselines under all delay settings and exhibits smaller performance degradation as delay increases. Right (bottom): Average execution time across all environments. Our method requires fewer steps and achieves faster task completion.
• Figure 3: Average completion progress. Progress is measured by discrete scores corresponding to the sub-tasks completed.
• Figure 4: Performance under delay injections on Grasp-Hard.