POE: Acoustic Soft Robotic Proprioception for Omnidirectional End-effectors

TL;DR

This paper presents POE, a tendon-driven soft finger equipped with six embedded microphones to enable acoustic proprioception. It introduces POE-M, a pipeline that first predicts physically grounded key-point displacements from acoustic signals and then reconstructs a watertight mesh via a smoothed ARAP energy optimization, leveraging key-point guidance for stability. Real- and simulation-based evaluations show that POE-M reduces the maximum Chamfer distance by $23.10\%$ and achieves an average Chamfer distance of $4.91~\mathrm{mm}$, outperforming baselines and providing complete shape estimates even from partial observations. The approach offers a scalable, occlusion-free alternative to vision-based proprioception with potential for real-time, multi-finger soft-robot pose estimation and control.

Abstract

Soft robotic shape estimation and proprioception are challenging because of soft robot's complex deformation behaviors and infinite degrees of freedom. A soft robot's continuously deforming body makes it difficult to integrate rigid sensors and to reliably estimate its shape. In this work, we present Proprioceptive Omnidirectional End-effector (POE), which has six embedded microphones across the tendon-driven soft robot's surface. We first introduce novel applications of previously proposed 3D reconstruction methods to acoustic signals from the microphones for soft robot shape proprioception. To improve the proprioception pipeline's training efficiency and model prediction consistency, we present POE-M. POE-M first predicts key point positions from the acoustic signal observations with the embedded microphone array. Then we utilize an energy-minimization method to reconstruct a physically admissible high-resolution mesh of POE given the estimated key points. We evaluate the mesh reconstruction module with simulated data and the full POE-M pipeline with real-world experiments. We demonstrate that POE-M's explicit guidance of the key points during the mesh reconstruction process provides robustness and stability to the pipeline with ablation studies. POE-M reduced the maximum Chamfer distance error by 23.10 % compared to the state-of-the-art end-to-end soft robot proprioception models and achieved 4.91 mm average Chamfer distance error during evaluation.

Figures (8)

• Figure 1: Overview of the proposed pipeline for Proprioceptive Omnidirectional End-effector (POE) acoustic soft robotic proprioception. We obtain acoustic signals from our novel tendon-driven soft robot POE with six embedded microphones (left). We then feed the extracted acoustic features into our proprioception modules that are able to reconstruct a high degree of freedom shape of POE (right).
• Figure 2: POE is a tendon-driven soft robot fabricated with a single molding step. Left column: The three-part mold is assembled with four metal rods (B) before the uncured silicone is poured in to create channels for tendons routing. The last part of the mold is then inserted to make the central conical cavity (C). After curing, the mold is disassembled to get the POE finger. Middle column: the finger secured into rest of POE assembly with two servo motors for tendon actuation. Right column: each servo controls POE movement in its perpendicular plane of bending, enabling POE to bend toward any direction.
• Figure 3: Spectogram visualizations from the six embedded microphones in two different shape configurations. The signal magnitudes change from one shape to another as we observe from the two rows of spectrograms. Time-varying features are ignored in the presented pipelines by averaging over the recording.
• Figure 4: Proposed POE-M Pipeline. First, concatenated acoustic feature vector from six microphones embedded in POE are used to predict new positions of the key points on POE's surface. POE-M uses the known correspondences between the key points and the vertices of POE surface mesh to iteratively fit the mesh to the predicted key points in a physically admissible manner.
• Figure 5: Evaluation of the As-Rigid-As-Possible (ARAP). Top row: SOFA FEM simulation environment schegg_sofagym_2023 and the generated meshes to be used for ARAP evaluation. Middle row: sensitivity study with varying $\lambda$ parameter. When $\lambda=0$ which corresponds to no neighboring edge rotation regularization, we observe undesirable surface artifacts. All nonzero $\lambda$ parameters removed the artifact effectively where $\lambda=5.0e-4$ yielded the lowest mesh reconstruction error. Bottom row: mesh updates over iterations. After each iteration, the mesh vertices that are not constrained are optimized to reduce the overall ARAP energy. We note that at around 30-50 iterations, the mesh converges.