Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Primary-Fine Decoupling for Action Generation in Robotic Imitation

Xiaohan Lei, Min Wang, Wengang Zhou, Xingyu Lu, Houqiang Li

TL;DR

Primary-Fine Decoupling for Action Generation (PF-DAG), a two-stage framework that decouples coarse action consistency from fine-grained variations, demonstrates that explicit mode-level decoupling enables both robust multi-modal modeling and reactive closed-loop control for robotic manipulation.

Abstract

Multi-modal distribution in robotic manipulation action sequences poses critical challenges for imitation learning. To this end, existing approaches often model the action space as either a discrete set of tokens or a continuous, latent-variable distribution. However, both approaches present trade-offs: some methods discretize actions into tokens and therefore lose fine-grained action variations, while others generate continuous actions in a single stage tend to produce unstable mode transitions. To address these limitations, we propose Primary-Fine Decoupling for Action Generation (PF-DAG), a two-stage framework that decouples coarse action consistency from fine-grained variations. First, we compress action chunks into a small set of discrete modes, enabling a lightweight policy to select consistent coarse modes and avoid mode bouncing. Second, a mode conditioned MeanFlow policy is learned to generate high-fidelity continuous actions. Theoretically, we prove PF-DAG's two-stage design achieves a strictly lower MSE bound than single-stage generative policies. Empirically, PF-DAG outperforms state-of-the-art baselines across 56 tasks from Adroit, DexArt, and MetaWorld benchmarks. It further generalizes to real-world tactile dexterous manipulation tasks. Our work demonstrates that explicit mode-level decoupling enables both robust multi-modal modeling and reactive closed-loop control for robotic manipulation.

Primary-Fine Decoupling for Action Generation in Robotic Imitation

TL;DR

Primary-Fine Decoupling for Action Generation (PF-DAG), a two-stage framework that decouples coarse action consistency from fine-grained variations, demonstrates that explicit mode-level decoupling enables both robust multi-modal modeling and reactive closed-loop control for robotic manipulation.

Abstract

Multi-modal distribution in robotic manipulation action sequences poses critical challenges for imitation learning. To this end, existing approaches often model the action space as either a discrete set of tokens or a continuous, latent-variable distribution. However, both approaches present trade-offs: some methods discretize actions into tokens and therefore lose fine-grained action variations, while others generate continuous actions in a single stage tend to produce unstable mode transitions. To address these limitations, we propose Primary-Fine Decoupling for Action Generation (PF-DAG), a two-stage framework that decouples coarse action consistency from fine-grained variations. First, we compress action chunks into a small set of discrete modes, enabling a lightweight policy to select consistent coarse modes and avoid mode bouncing. Second, a mode conditioned MeanFlow policy is learned to generate high-fidelity continuous actions. Theoretically, we prove PF-DAG's two-stage design achieves a strictly lower MSE bound than single-stage generative policies. Empirically, PF-DAG outperforms state-of-the-art baselines across 56 tasks from Adroit, DexArt, and MetaWorld benchmarks. It further generalizes to real-world tactile dexterous manipulation tasks. Our work demonstrates that explicit mode-level decoupling enables both robust multi-modal modeling and reactive closed-loop control for robotic manipulation.
Paper Structure (30 sections, 22 equations, 7 figures, 8 tables)

This paper contains 30 sections, 22 equations, 7 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Behavior Cloning
  4. Discrete Policy
  5. Generative Policy
  6. Hierarchical and Residual Policies
  7. PF-DAG Formulation and Design
  8. Closed-loop Action Sequence Prediction
  9. Observation Feature Extraction
  10. Primary Mode Policy and VQ-VAE
  11. Mode Conditioned MeanFlow Policy
  12. Theoretical analysis
  13. Experiments
  14. Simulation Evaluation
  15. Real World Evaluation
  16. ...and 15 more sections

Figures (7)

  • Figure 1: A 2D example illustrating multi-modal expert demonstrations and trajectories predicted by different imitation policies. Behavioral cloning predictions collapse into a single mean. Discrete Policy succeeds but introduces temporal discontinuities. Generative Policy bounces between mode 1 and 2. Our work predicts consistent and fine-grained trajectory.
  • Figure 2: Overview of our PF-DAG framework. The input observation features are extracted via Observation Feature Extraction and then fed to the Primary Mode Policy $\pi_1$. The GT action chunks are compressed into discrete primary modes using VQ-VAE and supervise $\pi_1$, which are only used in training stage. The Mode Conditioned MeanFlow Policy $\pi_2$ takes the selected primary mode $m$ and observation features as input, generating high-fidelity continuous actions.
  • Figure 3: Visual comparison of failure modes in baselines versus PF-DAG. Mode Collapse outputs "average" actions, while Mode Bouncing randomly switches between consecutive time steps.
  • Figure 4: Hardware and manipulated objects used in real world experiments.
  • Figure 5: Illustration of critical properties of PF-DAG. (a) Action chunks are projected to 2D via PCA, colored by their assigned primary mode. (b) PF-DAG’s one-step MeanFlow decoder achieves FPS comparable to 1-NFE DP3 while maintaining significantly higher success. (c) PF-DAG preserves high success even with short chunks by avoiding primary mode bouncing.
  • ...and 2 more figures