Direction Matters: Learning Force Direction Enables Sim-to-Real Contact-Rich Manipulation

TL;DR

This work employs a human-designed finite state machine based position/force controller in simulation to provide privileged guidance in simulation to provide privileged guidance, and provides theoretical analysis for stability and robustness to disturbances.

Abstract

Sim-to-real transfer for contact-rich manipulation remains challenging due to the inherent discrepancy in contact dynamics. While existing methods often rely on costly real-world data or utilize blind compliance through fixed controllers, we propose a framework that leverages expert-designed controller logic for transfer. Inspired by the success of privileged supervision in kinematic tasks, we employ a human-designed finite state machine based position/force controller in simulation to provide privileged guidance. The resulting policy is trained to predict the end-effector pose, contact state, and crucially the desired contact force direction. Unlike force magnitudes, which are highly sensitive to simulation inaccuracies, force directions encode high-level task geometry and remain robust across the sim-to-real gap. At deployment, these predictions configure a force-aware admittance controller. By combining the policy's directional intent with a constant, low-cost manually tuned force magnitude, the system generates adaptive, task-aligned compliance. This tuning is lightweight, typically requiring only a single scalar per contact state. We provide theoretical analysis for stability and robustness to disturbances. Experiments on four real-world tasks, i.e., microwave opening, peg-in-hole, whiteboard wiping, and door opening, demonstrate that our approach significantly outperforms strong baselines in both success rate and robustness. Videos are available at: https://yifei-y.github.io/project-pages/DirectionMatters/.

Figures (7)

• Figure 1: Compared to existing paradigms, our framework identifies force direction as a transferable signal that specifies task-relevant interaction intent and can be reliably learned in simulation, enabling force-aware policy training with pure simulation data and adaptive compliance in the real world.
• Figure 2: Overview of the proposed framework. (Left) In simulation, we implement an expert Finite State Machine based on privileged state to generate diverse demonstrations. (Middle) Identifying force direction and contact state as dynamics-invariant quantities that encode task intent, the policy is trained to predict these signals alongside poses using simulation data. (Right) In the real world, the policy outputs configure a force-aware admittance controller, which combines the predicted force direction with a manually specified magnitude to achieve adaptive, task-aligned compliance.
• Figure 3: The involved objects for four contact-rich manipulation tasks.
• Figure 4: Failure Mode Statistics. Baselines suffer from a high number and proportion of force-related failures, while our method significantly reduces both insufficient- and excessive-force failures.
• Figure 5: Force response during whiteboard wiping under nominal conditions (a) and height disturbances (b).