VQ-ACE: Efficient Policy Search for Dexterous Robotic Manipulation via Action Chunking Embedding

TL;DR

The results show that latent space sampling with MPC produces more human-like behavior in tasks such as Ball Rolling and Object Picking, leading to higher task success rates and reduced control costs, suggesting that VQ-ACE offers a scalable and effective solution for robotic manipulation tasks involving complex, high-dimensional state spaces.

Abstract

Dexterous robotic manipulation remains a significant challenge due to the high dimensionality and complexity of hand movements required for tasks like in-hand manipulation and object grasping. This paper addresses this issue by introducing Vector Quantized Action Chunking Embedding (VQ-ACE), a novel framework that compresses human hand motion into a quantized latent space, significantly reducing the action space's dimensionality while preserving key motion characteristics. By integrating VQ-ACE with both Model Predictive Control (MPC) and Reinforcement Learning (RL), we enable more efficient exploration and policy learning in dexterous manipulation tasks using a biomimetic robotic hand. Our results show that latent space sampling with MPC produces more human-like behavior in tasks such as Ball Rolling and Object Picking, leading to higher task success rates and reduced control costs. For RL, action chunking accelerates learning and improves exploration, demonstrated through faster convergence in tasks like cube stacking and in-hand cube reorientation. These findings suggest that VQ-ACE offers a scalable and effective solution for robotic manipulation tasks involving complex, high-dimensional state spaces, contributing to more natural and adaptable robotic systems.

Figures (7)

• Figure 1: We introduce , a method that learns a compact representation of complex human hand motion in a lower-dimensional space. For a 1-second action chunk with 11 , our approach encodes it into 5 tokens, each taking one of 4 possible discrete values. This learned latent space can be used in both sampling-based and , enabling control algorithms to search for optimal policies from an anthropomorphic prior.
• Figure 2: Samples of the tasks of data collection. The tasks covered everyday activities like bottle opening or keyboard typing.
• Figure 3: Architecture of Vector Quantized Action Chunking Embedding(VQ-ACE) and its applications in sampling based and . a. (Sec. \ref{['subsec:vqace']}). Conditioned on the current joint position, the encoder compresses the action sequence into a sequence of latent vectors. The latent vectors are quantized following the algorithm in Guyon2017vqvae. The decoder reconstructs the action sequence using both the joint position and quantized vectors. Causal masks are added to the decoder to allow time shifts in downstream applications. b. Latent sampling (Sec. \ref{['subsec:app_mjpc']}). The control signal is the sum of the nominal policy decoded from latents and a Gaussian noise spline. The Mujoco simulator evaluates all sampled control sequences and selects the sequence with the best cost for execution. New control sequences are generated by applying time-shifted noise to the best sequence. Gaussian noise is added to the spline, and latent vectors are randomly flipped. c. Action chunked (Sec. \ref{['subsec:app_rl']}). We augment the state and action spaces of a dexterous manipulation task with action chunk selection, which accumulates the agent's choices of action chunks and triggers the decoder. The decoder predicts the action chunk for the next steps, using the current joint position and the latent vector selected by the actor. The final action output is the sum of the action chunk and residual values.
• Figure 4: The first two rows show the execution of the Ball Rolling task. The screenshots are taken from with a time step of 0.2 seconds. In each figure, the left side shows the robotic hand in simulation, and the right side shows the orientation of the reference ball. We observe that our latent sampling generates more human-like behavior, with all fingers maintaining contact with the ball. In contrast, the baseline sampling-based MPC produces more arbitrary actions, as it merely selects the best control that drives the ball toward the target pose. The two bottom rows represent the Object Picking task. These snapshots are taken at intervals from 0s to 10s with a timestep of 2s. The transparent object represents the target pose. When the object is carried to a position within a threshold of the target, the task is considered successful, and both the object and the target pose are randomized for the next trial. In the third row (latent sampling ), the robot successfully grasps the object at 6 seconds and lifts it at 8 seconds. At 10s, the object and target positions are updated due to the robot's success. On the other hand, the baseline sampling-based MPC fails to grasp the object and gets stuck around the target without a successful attempt (last row).
• Figure 5: Cost and success rate in the Ball Rolling and Object Picking tasks. We compare our proposed latent sampling with the baseline predictive sampling and two ablations. The values in the histogram represent the mean, and the error bars represent the standard deviation of ten runs