TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance

TL;DR

Extensive experiments on five challenging contact-rich tasks, such as shoe lacing and chip handover, show that TouchGuide consistently and significantly outperforms state-of-the-art visuo-tactile policies.

Abstract

Fine-grained and contact-rich manipulation remain challenging for robots, largely due to the underutilization of tactile feedback. To address this, we introduce TouchGuide, a novel cross-policy visuo-tactile fusion paradigm that fuses modalities within a low-dimensional action space. Specifically, TouchGuide operates in two stages to guide a pre-trained diffusion or flow-matching visuomotor policy at inference time. First, the policy produces a coarse, visually-plausible action using only visual inputs during early sampling. Second, a task-specific Contact Physical Model (CPM) provides tactile guidance to steer and refine the action, ensuring it aligns with realistic physical contact conditions. Trained through contrastive learning on limited expert demonstrations, the CPM provides a tactile-informed feasibility score to steer the sampling process toward refined actions that satisfy physical contact constraints. Furthermore, to facilitate TouchGuide training with high-quality and cost-effective data, we introduce TacUMI, a data collection system. TacUMI achieves a favorable trade-off between precision and affordability; by leveraging rigid fingertips, it obtains direct tactile feedback, thereby enabling the collection of reliable tactile data. Extensive experiments on five challenging contact-rich tasks, such as shoe lacing and chip handover, show that TouchGuide consistently and significantly outperforms state-of-the-art visuo-tactile policies.

Figures (14)

• Figure 1: TacUMI is a low-cost yet high-precision handheld data collection system that provides direct tactile feedback through a rigid mechanical coupling. TouchGuide is a multi-modal fusion paradigm that steers a visuomotor policy via touch guidance during denoising or flow matching, producing actions that better adhere to contact physics without retraining the base policy.
• Figure 2: Overview of TacUMI data collection system. TacUMI (Collection-side) uses a Vive tracker for localization to obtain accurate end-effector poses, while the operator receives direct tactile feedback (Left). During policy inference, we use an execution-side device that is structurally identical to the collection-side TacUMI, coupled to different robot arms via an adapter.
• Figure 3: Overview of TouchGuide framework. (a) The architecture of the task-specific Contact Physical Model (CPM). (b) During inference, the CPM serves as an external model that steers the base policy’s action generation within the sampling process using a feasibility score. (c) In action space, TouchGuide can be viewed as a form of contact-physics steering that steers the policy distribution toward the real distribution.
• Figure 4: Comparison of policy distributions in action space.
• Figure 5: Five experiment tasks including Shoe Lacing, Chip Handover, Cucumber Peeling, Vase Wiping, and Lock Opening.

Theorems & Definitions (3)

• Proposition 1
• Remark 1
• proof