Proprioceptive Learning with Soft Polyhedral Networks

TL;DR

The paper tackles the problem of achieving robust proprioception in soft, low-cost robotic grippers. It introduces Soft Polyhedral Networks with embedded vision to enable passive omni-directional adaptation and vision-based proprioception, complemented by Sim2Real kinesthesia learning from FEM data to predict real-time 3D deformation. A visual force learning pipeline incorporates viscoelastic effects to estimate 6D forces/torques with high accuracy, while experiments demonstrate durable, high-performance grasping, tactile reconstruction, and safe impact absorption over more than 1 million cycles. The approach offers a scalable, inexpensive pathway to integrate proprioception and tactile sensing into soft robotic manipulation, with implications for adaptive grasping, soft manipulation, and human-robot interaction.

Abstract

Proprioception is the "sixth sense" that detects limb postures with motor neurons. It requires a natural integration between the musculoskeletal systems and sensory receptors, which is challenging among modern robots that aim for lightweight, adaptive, and sensitive designs at a low cost. Here, we present the Soft Polyhedral Network with an embedded vision for physical interactions, capable of adaptive kinesthesia and viscoelastic proprioception by learning kinetic features. This design enables passive adaptations to omni-directional interactions, visually captured by a miniature high-speed motion tracking system embedded inside for proprioceptive learning. The results show that the soft network can infer real-time 6D forces and torques with accuracies of 0.25/0.24/0.35 N and 0.025/0.034/0.006 Nm in dynamic interactions. We also incorporate viscoelasticity in proprioception during static adaptation by adding a creep and relaxation modifier to refine the predicted results. The proposed soft network combines simplicity in design, omni-adaptation, and proprioceptive sensing with high accuracy, making it a versatile solution for robotics at a low cost with more than 1 million use cycles for tasks such as sensitive and competitive grasping, and touch-based geometry reconstruction. This study offers new insights into vision-based proprioception for soft robots in adaptive grasping, soft manipulation, and human-robot interaction.

Figures (10)

• Figure 1: Soft Polyhedral Network design with embedded vision. (A) A generic design process applicable to all polyhedrons, (i) starting with removing all faces and replacing all edges with beam structures made from soft materials, then (ii) adding layers inside with the flexure joints, resulting in (iii) a class of soft networks that are geometrically adaptive to external interactions. (B) An enhanced version of the Soft Polyhedral Network with a primary interaction face (marked in pink) and a secondary interaction face (marked in blue). The primary face has an extended contact area with a trapezoid frame, and the secondary face enables adaption in 3D. (C) Exploded view for vision integration by mounting the soft network on top of a base frame housing a high-speed miniature camera, capturing the soft network's 6D motion during adaptation by tracking an ArUco marker attached inside. (D) The pipeline for proprioceptive learning when using the Soft Polyhedral Network as fingers of a common gripper system. The camera captures the spatial deformation of the soft network by tracking the ArUco marker's 6D movement. We feed pose and velocity inputs to a neural network to infer 6D forces and torques as the output, which can be further processed to estimate the gripping and shear forces and fed to the robot control loop for reactive object manipulation based on the friction cone model.
• Figure 2: Experiment setup for measuring stiffness.
• Figure 3: Stiffness distribution of the Soft Polyhedral Network measured using the test rig and FEM.
• Figure 4: Learning adaptive kinesthesia with Sim2Real proprioception. (A) Test for the soft network's passive adaptation by placing a roller at four different locations (i)$\sim$(iv) on the primary interaction face. The roller moves towards an equilibrium area marked in orange dashed lines. (B) FEM simulations of the primary interaction face's adaptive deformation when applying 2$\sim$12 N forces at four initial contact locations marked with arrows. Note that the maximum deformation always occurs within an equilibrium region marked in light blue. (C) Measurement of the adaptive factor $\kappa$ for the primary and secondary interaction faces. Both faces exhibit passive adaptation with $\kappa$ maximizing near $L_3$, resulting in an enclosed adaptation of the soft network upon external compression. Note that the adaptive capability in the primary interaction face is greater than the secondary one. (D) After (i) collecting FEM simulation data of the soft network under external compressions at various angles and magnitudes, (ii) we train a Sim2Real multi-layer perceptron (MLP) to reproduce the spatial movement of 26 key points on the soft network. (iii) When deployed to the soft network prototype, the MLP predictions align well with observations in free-standing, pushing, and twisting scenarios.
• Figure 5: Mean error distribution of Sim2Real learning for adaptive kinesthesia. The shaded area indicates one standard deviation of the average positional error.