RFS: Reinforcement learning with Residual flow steering for dexterous manipulation

TL;DR

RFS tackles the generalization gap in pretrained multimodal imitation policies for dexterous manipulation by introducing Residual Flow Steering, which jointly modulates a flow-matching base policy through latent-noise steering and an affine residual action. The method integrates input modulation over latent variables with output modulation via a residual, enabling global behavioral changes while preserving fine-grained control, all without updating the base model parameters. Empirically, RFS yields strong sim-to-real transfer by first pretraining in simulation on flow-matching policies, then performing offline real-world fine-tuning with a TD3+BC objective, and demonstrates superior performance across six dexterous tasks in simulation and robust adaptation to unseen real-world objects. The approach achieves data-efficient adaptation, leveraging limited demonstrations and corrective feedback to improve reliability in complex manipulation tasks, with potential impact on deploying generative imitation policies in real robots.

Abstract

Imitation learning has emerged as an effective approach for bootstrapping sequential decision-making in robotics, achieving strong performance even in high-dimensional dexterous manipulation tasks. Recent behavior cloning methods further leverage expressive generative models, such as diffusion models and flow matching, to represent multimodal action distributions. However, policies pretrained in this manner often exhibit limited generalization and require additional fine-tuning to achieve robust performance at deployment time. Such adaptation must preserve the global exploration benefits of pretraining while enabling rapid correction of local execution errors.We propose \emph{Residual Flow Steering} (RFS), a data-efficient reinforcement learning framework for adapting pretrained generative policies. RFS steers a pretrained flow-matching policy by jointly optimizing a residual action and a latent noise distribution, enabling complementary forms of exploration: local refinement through residual corrections and global exploration through latent-space modulation. This design allows efficient adaptation while retaining the expressive structure of the pretrained policy.We demonstrate the effectiveness of RFS on dexterous manipulation tasks, showing efficient fine-tuning both in simulation and in real-world settings when adapting pretrained base policies.Project website:https://weirdlabuw.github.io/rfs.

Figures (9)

• Figure 1: Residual Flow Steering (RFS). Given a state $s$, the RFS policy $\pi_{\text{RFS}}$ outputs a latent flow variable $w_0$ and a residual action $a_r$, which jointly steer a pretrained base policy $\pi_{\text{FM}}$ to produce the final action $a_b + a_r$. RFS enables both global mode shifting and fine-grained residual correction, allowing the policy to expand beyond the demonstration data manifold.
• Figure 2: Overview of the sim-to-real Residual Flow Steering (RFS) pipeline. (1) VR teleoperation is used to collect demonstrations across multiple manipulation tasks to train task-specific flow-matching base policies. (2) In simulation, the RFS policy $\pi_{\text{RFS}}$ is fine-tuned on top of each base policy and distilled into task-specific visuomotor policies to improve sim-to-real transfer. (3) During zero-shot real-world deployment, human corrective actions correct execution failures such as unstable grasps and misplacement. (4) These corrected transitions are used for offline fine-tuning of $\pi_{\text{RFS}}$ on a Franka--Leap Hand system, improving real-world grasping and pick-and-place performance.
• Figure 2: Performance on seen objects. Values report mean success rate with 95% confidence interval across trials.
• Figure 3: Simulation and real objects used for dexterous grasping and pick & place.
• Figure 4: Representative rollouts for the dexterous manipulation tasks. From top to bottom: Packing, Push-to-Grasp, Packing and Stacking.