Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Temporal Action Selection for Action Chunking

Yueyang Weng, Xiaopeng Zhang, Yongjin Mu, Yingcong Zhu, Yanjie Li, Qi Liu

TL;DR

The paper addresses the reactivity-versus-consistency trade-off in action chunking for learning from demonstration by introducing Temporal Action Selector (TAS), which caches action chunks from multiple timesteps and selects actions via a lightweight latent-space, cosine-similarity-based selector. TAS is trained with online RL (PPO) using sparse rewards and a coherence penalty, and can be integrated with residual RL in either a frozen or jointly optimized mode, improving base-policy performance without catastrophic forgetting. Across PushT and FurnitureBench tasks with diverse base policies and noise settings, TAS yields substantial success-rate gains over baselines and demonstrates strong sim-to-real transfer in real-world one_leg experiments. The approach balances reactivity, decision consistency, and motion coherence and shows significant potential for enhancing RL training efficiency and deployment in robotic manipulation.

Abstract

Action chunking is a widely adopted approach in Learning from Demonstration (LfD). By modeling multi-step action chunks rather than single-step actions, action chunking significantly enhances modeling capabilities for human expert policies. However, the reduced decision frequency restricts the utilization of recent observations, degrading reactivity - particularly evident in the inadequate adaptation to sensor noise and dynamic environmental changes. Existing efforts to address this issue have primarily resorted to trading off reactivity against decision consistency, without achieving both. To address this limitation, we propose a novel algorithm, Temporal Action Selector (TAS), which caches predicted action chunks from multiple timesteps and dynamically selects the optimal action through a lightweight selector network. TAS achieves balanced optimization across three critical dimensions: reactivity, decision consistency, and motion coherence. Experiments across multiple tasks with diverse base policies show that TAS significantly improves success rates - yielding an absolute gain of up to 73.3%. Furthermore, integrating TAS as a base policy with residual reinforcement learning (RL) substantially enhances training efficiency and elevates the performance plateau. Experiments in both simulation and physical robots confirm the method's efficacy.

Temporal Action Selection for Action Chunking

TL;DR

The paper addresses the reactivity-versus-consistency trade-off in action chunking for learning from demonstration by introducing Temporal Action Selector (TAS), which caches action chunks from multiple timesteps and selects actions via a lightweight latent-space, cosine-similarity-based selector. TAS is trained with online RL (PPO) using sparse rewards and a coherence penalty, and can be integrated with residual RL in either a frozen or jointly optimized mode, improving base-policy performance without catastrophic forgetting. Across PushT and FurnitureBench tasks with diverse base policies and noise settings, TAS yields substantial success-rate gains over baselines and demonstrates strong sim-to-real transfer in real-world one_leg experiments. The approach balances reactivity, decision consistency, and motion coherence and shows significant potential for enhancing RL training efficiency and deployment in robotic manipulation.

Abstract

Action chunking is a widely adopted approach in Learning from Demonstration (LfD). By modeling multi-step action chunks rather than single-step actions, action chunking significantly enhances modeling capabilities for human expert policies. However, the reduced decision frequency restricts the utilization of recent observations, degrading reactivity - particularly evident in the inadequate adaptation to sensor noise and dynamic environmental changes. Existing efforts to address this issue have primarily resorted to trading off reactivity against decision consistency, without achieving both. To address this limitation, we propose a novel algorithm, Temporal Action Selector (TAS), which caches predicted action chunks from multiple timesteps and dynamically selects the optimal action through a lightweight selector network. TAS achieves balanced optimization across three critical dimensions: reactivity, decision consistency, and motion coherence. Experiments across multiple tasks with diverse base policies show that TAS significantly improves success rates - yielding an absolute gain of up to 73.3%. Furthermore, integrating TAS as a base policy with residual reinforcement learning (RL) substantially enhances training efficiency and elevates the performance plateau. Experiments in both simulation and physical robots confirm the method's efficacy.

Paper Structure

This paper contains 15 sections, 7 equations, 10 figures, 5 tables.

Table of Contents

  1. INTRODUCTION
  2. PROBLEM FORMULATION
  3. METHOD
  4. Temporal Action Selection Architecture
  5. Online Reinforcement Learning
  6. Integration with Residual Reinforcement Learning
  7. EXPERIMENTS
  8. Tasks and Base Policies
  9. Delay Policies
  10. Performance of TAS
  11. Comparative Analysis of Selector Network Architectures
  12. Integration with Residual Reinforcement Learning
  13. The Necessity of Action Caching
  14. Real World Experiments
  15. CONCLUSIONS AND FUTURE WORK

Figures (10)

  • Figure 1: Classical Action Chunking Illustration zhao2023learning.
  • Figure 2: Overview of TAS. The base policy generates candidate actions from observations, followed by the selector network choosing optimal actions via latent-space similarity.
  • Figure 3: Alternative selector architectures.
  • Figure 4: The simulated environments where we evaluated TAS: PushT taskchi2023diffusionpolicy and one_leg, lamp, round_table tasks from the FurnitureBenchheo2023furniturebench task suite
  • Figure 5: Performance across Action Delays. Evaluation metrics under four configurations reveal that optimal policy performance emerges at non-zero delays.
  • ...and 5 more figures