Highly Deformable Proprioceptive Membrane for Real-Time 3D Shape Reconstruction

TL;DR

This work targets real-time, vision-robust 3D shape reconstruction by embedding a highly deformable optical waveguide membrane in soft robotics. A two-stage learning framework—a point-cloud autoencoder to capture a shape prior and a PD-to-latent regressor to map optical signals to that latent space—translates sparse, intensity-based measurements into dense 3D geometry. The 140×140 mm prototype achieves a 1.307 mm average Chamfer distance at 90 Hz while withstanding large deformations up to 25 mm, and SAGE-based analysis provides concrete guidance for sensor layout and minimization. The approach offers EMI-robust, scalable, low-profile proprioceptive sensing with potential for broad deployment in deformable robots and tactile-enabled systems.

Abstract

Reconstructing the three-dimensional (3D) geometry of object surfaces is essential for robot perception, yet vision-based approaches are generally unreliable under low illumination or occlusion. This limitation motivates the design of a proprioceptive membrane that conforms to the surface of interest and infers 3D geometry by reconstructing its own deformation. Conventional shape-aware membranes typically rely on resistive, capacitive, or magneto-sensitive mechanisms. However, these methods often encounter challenges such as structural complexity, limited compliance during large-scale deformation, and susceptibility to electromagnetic interference. This work presents a soft, flexible, and stretchable proprioceptive silicone membrane based on optical waveguide sensing. The membrane sensor integrates edge-mounted LEDs and centrally distributed photodiodes (PDs), interconnected via liquid-metal traces embedded within a multilayer elastomeric composite. Rich deformation-dependent light intensity signals are decoded by a data-driven model to recover the membrane geometry as a 3D point cloud. On a customized 140 mm square membrane, real-time reconstruction of large-scale out-of-plane deformation is achieved at 90 Hz with an average reconstruction error of 1.3 mm, measured by Chamfer distance, while maintaining accuracy for indentations up to 25 mm. The proposed framework provides a scalable, robust, and low-profile solution for global shape perception in deformable robotic systems.

Figures (7)

• Figure 1: Design of highly deformable optical waveguide sensor for surface shape reconstruction.(A). Prototype system of the sensorized optical waveguide membrane, together with an example reconstruction: top view and cross sections, compare the ground-truth shape with the reconstructed point cloud. (B). Overview of the sensing principle: edge-mounted LEDs inject light into the transparent core of the waveguide, and embedded photodiodes measure the deformation-induced change in light intensity. (C).$\ell$ LEDs are strobed one at a time, and $p$ photodiodes are sampled simultaneously for each LED. A full scan of all LEDs forms a $p\times \ell$ measurement matrix, which is then fed into a neural network to reconstruct the surface shape point cloud. (D). Photograph of the sensor under representative deformations (bend, twist, and stretch), which shows the potential of operation under highly-deformed scenarios.
• Figure 2: Hardware architecture and data acquisition system.(A). Exploded CAD view of the optical waveguide sensor showing internal layer stack, photodiode and LED array, stretchable OGaIn interconnects, and flexible flat cable interface. Flexible PCB connectors are designed to fold upward to mount LEDs vertically in the waveguide. (B). Photograph of the sensor prototype with the electrical wiring overlaid. An STM32 microcontroller sequences the LEDs and coordinates photodiode sampling via the external ADC, producing a complete measurement frame by concatenating responses across all LED-photodiode pairs.
• Figure 3: Manufacturing process of the optical waveguide sensor.(A). Spin coat the bottom cladding (black) layer in a 3D-printed mold. (B). Spin coat the bottom reflective (white) layer on top of the black layer. (C). Place a piece of VHB-4905 sheet on the cured white layer and seal the side with elastomer (Ecoflex 00-45). (D). Cut circuit traces on the sticker paper of the VHB using a laser cutter. (E). Paint liquid metal (OGaIn) on the sticker paper to fill the traces. (F). Peel the sticker paper and leave the OGaIn traces on the VHB. (G). Place LEDs and PDs flexible PCB connectors on the circuit and test the circuit. (H). Cast the transparent core of the waveguide. (I). Cast the top reflective layer. (J). Cast the top cladding layer, and demold the entire sensor after it is fully cured.
• Figure 4: Data-driven model for surface shape reconstruction(A). Stage 1: Autoencoder pre-training. A point-cloud autoencoder is trained using ground-truth surface point clouds. The encoder compresses the point cloud (shape: $(M_{\mathrm{gt}}, 3)$) into a latent vector (shape: $(L, )$), and the decoder reconstructs the point cloud (shape: $(M_{\mathrm{pr}}, 3)$) from the latent space. (B). Optical signal to shape inference. Normalized photodiode measurements (shape: $(150, 0)$) are mapped to a latent vector using a multi-layer perceptron, and the pre-trained point-cloud decoder generates the predicted point cloud (shape: $(M_{\mathrm{pr}}, 3)$).
• Figure 5: Characterization of the waveguide sensor.(A). Gravity-loaded bending setup: Left: wall angle controls the sensor curvature; Right: top-view layout of LEDs and photodiodes (PDs) with the bending direction indicated. (B). A comparison of PD responses versus wall angle between two representative LED-PD pairs. The pair aligned with the bending direction (LED B--PD3, green) shows a strong intensity change, while the transverse pair (LED A--PD3, pink) remains largely stable. (C). Single feature (LED/PD) importance evaluation based on SAGE values and the relation with physical locations. (D). Reconstruction performance under progressive feature inclusion. Top: PDs are added one-by-one with all LEDs enabled; Bottom: LEDs are added one-by-one with all PDs enabled. Curves compare different inclusion orders (Natural, SAGE-ranked, and reverse SAGE). (E). Sample reconstruction error shown as a heatmap of nearest neighbor (NN) distance of representative reduced models with different numbers of LEDs.