FreqPolicy: Frequency Autoregressive Visuomotor Policy with Continuous Tokens

TL;DR

A novel paradigm for visuomotor policy learning that progressively models hierarchical frequency components is proposed, showcasing the potential of a frequency-domain autoregressive framework with continuous tokens for generalized robotic manipulation.

Abstract

Learning effective visuomotor policies for robotic manipulation is challenging, as it requires generating precise actions while maintaining computational efficiency. Existing methods remain unsatisfactory due to inherent limitations in the essential action representation and the basic network architectures. We observe that representing actions in the frequency domain captures the structured nature of motion more effectively: low-frequency components reflect global movement patterns, while high-frequency components encode fine local details. Additionally, robotic manipulation tasks of varying complexity demand different levels of modeling precision across these frequency bands. Motivated by this, we propose a novel paradigm for visuomotor policy learning that progressively models hierarchical frequency components. To further enhance precision, we introduce continuous latent representations that maintain smoothness and continuity in the action space. Extensive experiments across diverse 2D and 3D robotic manipulation benchmarks demonstrate that our approach outperforms existing methods in both accuracy and efficiency, showcasing the potential of a frequency-domain autoregressive framework with continuous tokens for generalized robotic manipulation.Code is available at https://github.com/4DVLab/Freqpolicy

Figures (12)

• Figure 1: Action signals reconstructed across different frequency bands; see Appendix for energy details.
• Figure 2: Pipeline of FreqPolicy showing both training (a) and inference (b) procedures. We first transform action trajectories into the frequency domain via DCT, and then learns latent codes for different frequency level actions using FreqPolicy, and reconstructs actions through masked prediction and a diffusion-based decoder. This enables robust, frequency-aware, and high-fidelity robotic action generation.
• Figure 3: (a)Heat maps of frequency band energy across action dimensions for different tasks. The top row shows Adroit rajeswaran2017dapg tasks with high-dimensional actions (26 dimensions), while the bottom row presents Robomimic mandlekar2021robomimic tasks with low-dimensional actions (10 dimensions). (b)Success rate of actions reconstructed with varying frequency ratios. We reconstruct action sequences using different proportions of frequency components and evaluate their success rates on the original tasks.
• Figure 4: Pareto Analysis on Adroit benchmark. The x-axis represents inference time and the y-axis indicates task success rate.
• Figure 5: Real-World Experiments on Robotic Handover Task.The robotic hand stably receives an object from a human subject during real-world testing.