KineSoft: Learning Proprioceptive Manipulation Policies with Soft Robot Hands

TL;DR

KineSoft tackles dexterous manipulation with underactuated soft hands by turning their inherent compliance into an advantage for learning from kinesthetic demonstrations. The approach combines a mesh-based proprioceptive shape estimation from embedded strain sensors, a diffusion-based imitation policy, and a shape-conditioned low-level controller to ground high-level trajectories in geometric deformations. Experiments show accurate shape estimation ($1.92$ mm error), precise shape tracking ($3.29$ mm), and strong task performance across six in-hand tasks, including a Bottle Unscrewing success rate of $85\%$ and Lid Flicking of $100\%$, outperforming baseline strain-based methods. The work demonstrates effective sim-to-real transfer and highlights the importance of intermediate shape-based representations to bridge demonstration and execution in soft robots, with potential extensions to tactile sensing and interface tooling.

Abstract

Underactuated soft robot hands offer inherent safety and adaptability advantages over rigid systems, but developing dexterous manipulation skills remains challenging. While imitation learning shows promise for complex manipulation tasks, traditional approaches struggle with soft systems due to demonstration collection challenges and ineffective state representations. We present KineSoft, a framework enabling direct kinesthetic teaching of soft robotic hands by leveraging their natural compliance as a skill teaching advantage rather than only as a control challenge. KineSoft makes two key contributions: (1) an internal strain sensing array providing occlusion-free proprioceptive shape estimation, and (2) a shape-based imitation learning framework that uses proprioceptive feedback with a low-level shape-conditioned controller to ground diffusion-based policies. This enables human demonstrators to physically guide the robot while the system learns to associate proprioceptive patterns with successful manipulation strategies. We validate KineSoft through physical experiments, demonstrating superior shape estimation accuracy compared to baseline methods, precise shape-trajectory tracking, and higher task success rates compared to baseline imitation learning approaches.

Figures (9)

• Figure 1: KineSoft is a framework for learning from kinesthetic demonstration, enabling free-shaped soft end-effectors to perform dexterous manipulation. Three key components are: 1) a proprioceptive model for high-fidelity shape estimation, 2) diffusion-based imitation learning for predicting the changes in shape and end-effector poses, and 3) a shape-conditioned controller that allows the soft hand to track given shape trajectories.
• Figure 2: Proprioception network.: Network architecture for mesh shape estimation of the soft fingers. : Results of domain alignment iterations where the loss converged after 200 iterations.
• Figure 3: Demonstration and KineSoft Rollout for Bottle Unscrewing Task.
• Figure 4: Tasks. We evaluate the performance of the shape-based KineSoft policy across six manipulation tasks. These tasks highlight the advantages of soft robotic hands in contact-rich and delicate object interaction. Red arrows indicate actuation directions for Servo 1; blue arrows indicate actuation directions for Servo 2.
• Figure 5: Tasks