Autoregressive Action Sequence Learning for Robotic Manipulation

TL;DR

This work tackles universal robotic manipulation by reframing actions as autoregressive sequences and introducing a Chunking Causal Transformer (CCT) to predict variable-length action chunks. Built atop CCT, the Autoregressive Policy (ARP) generates hybrid action sequences that combine discrete, continuous, and perceptual actions, trained with attention interleaving and teacher-forcing. Across Push-T, ALOHA, and RLBench, ARP matches or surpasses environment-specific state-of-the-art methods while reducing computation and parameter counts, and it demonstrates practical viability with real-robot nut-tightening. The results suggest chunking autoregression as a robust backbone for diverse robotic control and point to future directions in planning, interactive learning, and universal action languages.

Abstract

Designing a universal policy architecture that performs well across diverse robots and task configurations remains a key challenge. In this work, we address this by representing robot actions as sequential data and generating actions through autoregressive sequence modeling. Existing autoregressive architectures generate end-effector waypoints sequentially as word tokens in language modeling, which are limited to low-frequency control tasks. Unlike language, robot actions are heterogeneous and often include continuous values -- such as joint positions, 2D pixel coordinates, and end-effector poses -- which are not easily suited for language-based modeling. Based on this insight, we introduce a straightforward enhancement: we extend causal transformers' single-token prediction to support predicting a variable number of tokens in a single step through our Chunking Causal Transformer (CCT). This enhancement enables robust performance across diverse tasks of various control frequencies, greater efficiency by having fewer autoregression steps, and lead to a hybrid action sequence design by mixing different types of actions and using a different chunk size for each action type. Based on CCT, we propose the Autoregressive Policy (ARP) architecture, which solves manipulation tasks by generating hybrid action sequences. We evaluate ARP across diverse robotic manipulation environments, including Push-T, ALOHA, and RLBench, and show that ARP, as a universal architecture, matches or outperforms the environment-specific state-of-the-art in all tested benchmarks, while being more efficient in computation and parameter sizes. Videos of our real robot demonstrations, all source code and the pretrained models of ARP can be found at http://github.com/mlzxy/arp.

Figures (22)

• Figure 1: Existing Works versus Our Autoregressive Policy. Action chunking models (left) predict all action tokens in a single step zhao2023alohakim2024openvlawu2023gr1. Standard autoregression models (middle) generate one action token in each step, which is inefficient and unsuitable for high-frequency control tasks chen2021decisionTjanner2021trajTreed2022gatojiang2022vimali2024manipllm. Our proposed chunking autoregression (right) generates a chunk of variable number of action tokens per step, offering greater efficiency, strong performance across diverse tasks, and flexibility in designing hybrid action sequences. We compare the performance of these three action prediction strategies in Figure \ref{['tab:generation-mode']}. Note all models use Model Predictive Control to predict $L$ actions, execute them, update the observation, and then predict actions again. Autoregressive generation is performed without executing actions or changing the current observation.
• Figure 2: Overview of the simulation environments. We evaluate our method on Push-T, ALOHA, and RLBench, three task suites with significantly different properties and requirements. Push-T chi2023diffp requires many steps to complete (long horizon) and where same sub-goals can be reached in various ways (multi-modality). ALOHA zhao2023aloha has a high-dimensional action space (14 joints of two robot arms), a high control frequency (50Hz), and a short time limit (8 secs). RLBench james2020rlbench has only the gripper pose as action but contains 18 different language-conditioned tasks.
• Figure 3: Learned Action Sequences. In Push-T, our model predicts a sequence of high-level waypoints, followed by a sequence of low-level positions that connect the waypoints together and form the pushing trajectory, analogous to hierarchical planning hafner2022hiera-plan. In ALOHA, we predict the joint values and then end-effector waypoints conditioned on the joint values, a process akin to forward kinematics kucuk2006robot. We bypass the waypoint generation during inference. In RLBench, we predict the target end-effector's position first, then gripper rotation and state in that position. For our real robot experiment, we define a set of primitive actions, as detailed in section \ref{['sec:real-robot']}. We predict the action type and then continuous values of that action.
• Figure 4: Autoregressive Policy Architecture. A sequence of past actions and a chunk of empty tokens are concatenated and projected into embeddings. Empty tokens correspond to future actions, which are unknown and need to be predicted. These embeddings are fed into our Chunking Causal Transformer (CCT) along with the vision features of the current observation. CCT alternates between self-attention within the sequence embeddings and cross-attention with the vision features. Self-attention is causal for the input actions and bidirectional among the empty tokens. Distributions of future actions are decoded from the updated embeddings of the empty tokens.
• Figure 5: Embeddings for Discrete, Continuous, and Pixel-coordinate Actions. Discrete actions are embedded by a simple table lookup on a weight matrix. Continuous actions are embedded with a linear layer. Pixel-coordinate actions are embedded by retrieving the point-wise features at the coordinates on the visual feature maps.