Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Harnessing Deep Learning of Point Clouds for Inverse Control of 3D Shape Morphing

Jue Wang, Dhirodaatto Sarkar, Jiaqi Suo, Alex Chortos

TL;DR

This work tackles inverse control for 3D shape morphing devices formed from soft actuator arrays by expressing deformations as 3D point clouds and learning a direct mapping to actuator inputs. It introduces Shape Morphing Net (SMNet), a regression framework that combines Kernel Point Convolution and PointNet++ to convert point-cloud representations of target shapes into high-dimensional control vectors, trained entirely on Finite Element Analysis (FEA) data. SMNet achieves state-of-the-art accuracy across ionic, thermal, and pneumatic actuators in both 2D and 3D PSM, with R2 scores reaching up to 0.9768 for ionic 2D and robust performance for 3D devices, enabling accurate inverse control and real-time capable operation. The method offers a universal, model-free framework for rapid development of 3D shape morphing systems, with potential impact on soft robotics, human–machine interfaces, and biomimetic applications, while highlighting avenues for bridging to physical hardware and interior-point data.

Abstract

Shape-morphing devices, a crucial branch in soft robotics, hold significant application value in areas like human-machine interfaces, biomimetic robotics, and tools for interacting with biological systems. To achieve three-dimensional (3D) programmable shape morphing (PSM), the deployment of array-based actuators is essential. However, a critical knowledge gap impeding the development of 3D PSM is the challenge of controlling the complex systems formed by these soft actuator arrays. This study introduces a novel approach, for the first time, representing the configuration of shape morphing devices using point cloud data and employing deep learning to map these configurations to control inputs. We propose Shape Morphing Net (SMNet), a method that realizes the regression from point cloud data to high-dimensional continuous vectors. Applied to previous 2D PSM actuator arrays, SMNet significantly enhances control precision from 82.23% to 97.68%. Further, we extend its application to 3D PSM devices with three different actuator mechanisms, demonstrating the universal applicability of SMNet to the control of 3D shape morphing technologies. In our demonstrations, we confirm the efficacy of inverse control, where 3D PSM devices successfully replicate target shapes. These shapes are obtained either through 3D scanning of physical objects or via 3D modeling software. The results show that within the deformable range of 3D PSM devices, accurate reproduction of the desired shapes is achievable. The findings of this research represent a substantial advancement in soft robotics, particularly for applications demanding intricate 3D shape transformations, and establish a foundational framework for future developments in the field.

Harnessing Deep Learning of Point Clouds for Inverse Control of 3D Shape Morphing

TL;DR

This work tackles inverse control for 3D shape morphing devices formed from soft actuator arrays by expressing deformations as 3D point clouds and learning a direct mapping to actuator inputs. It introduces Shape Morphing Net (SMNet), a regression framework that combines Kernel Point Convolution and PointNet++ to convert point-cloud representations of target shapes into high-dimensional control vectors, trained entirely on Finite Element Analysis (FEA) data. SMNet achieves state-of-the-art accuracy across ionic, thermal, and pneumatic actuators in both 2D and 3D PSM, with R2 scores reaching up to 0.9768 for ionic 2D and robust performance for 3D devices, enabling accurate inverse control and real-time capable operation. The method offers a universal, model-free framework for rapid development of 3D shape morphing systems, with potential impact on soft robotics, human–machine interfaces, and biomimetic applications, while highlighting avenues for bridging to physical hardware and interior-point data.

Abstract

Shape-morphing devices, a crucial branch in soft robotics, hold significant application value in areas like human-machine interfaces, biomimetic robotics, and tools for interacting with biological systems. To achieve three-dimensional (3D) programmable shape morphing (PSM), the deployment of array-based actuators is essential. However, a critical knowledge gap impeding the development of 3D PSM is the challenge of controlling the complex systems formed by these soft actuator arrays. This study introduces a novel approach, for the first time, representing the configuration of shape morphing devices using point cloud data and employing deep learning to map these configurations to control inputs. We propose Shape Morphing Net (SMNet), a method that realizes the regression from point cloud data to high-dimensional continuous vectors. Applied to previous 2D PSM actuator arrays, SMNet significantly enhances control precision from 82.23% to 97.68%. Further, we extend its application to 3D PSM devices with three different actuator mechanisms, demonstrating the universal applicability of SMNet to the control of 3D shape morphing technologies. In our demonstrations, we confirm the efficacy of inverse control, where 3D PSM devices successfully replicate target shapes. These shapes are obtained either through 3D scanning of physical objects or via 3D modeling software. The results show that within the deformable range of 3D PSM devices, accurate reproduction of the desired shapes is achievable. The findings of this research represent a substantial advancement in soft robotics, particularly for applications demanding intricate 3D shape transformations, and establish a foundational framework for future developments in the field.
Paper Structure (20 sections, 9 equations, 7 figures, 8 tables, 1 algorithm)

This paper contains 20 sections, 9 equations, 7 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Results
  3. Data collection and preprocessing of point cloud data
  4. Architecture of SMNet
  5. Model performance
  6. Ionic 2D low-profile PSM & 3D PSM
  7. Pneumatic 3D PSM & thermal 3D PSM
  8. Inverse demos for 3D PSM model-free control
  9. Discussion
  10. Methods
  11. FEA simulation of actuation mechanisms
  12. Principle of SMNet
  13. Training setting-ups
  14. The evaluation methods of the similarity of point clouds
  15. Complexity of different shapes
  16. ...and 5 more sections

Figures (7)

  • Figure 1: A universal method for controlling 3D shape morphing devices by mapping the point cloud of the deformed configuration with the control inputs of devices. a, Utilizing the point cloud to express the configurations of shape morphing devices, using an octopus to symbolize a shape morphing device. b, The rendering for 3D shape morphing devices based on ionic actuator arrays. c, The rendering for 3D shape morphing devices based on thermal actuator arrays. d, The rendering for 3D shape morphing devices based on pneumatic actuator arrays. e, the demonstration procedures by which 3D shape morphing devices reproduce the shape of physical objects in simulations.
  • Figure 2: The framework of mapping point cloud from simulation results with the control inputs by using SMNet. a, The procedures of extracting point cloud data from simulations. b, The downsampling strategy for point cloud data: including grid average downsampling to avoid the point concentration and random downsampling to ensure the number of points is the same. c, The point cloud rotation and normalization for training requirements. d, The architecture of SMNet for regression problems.
  • Figure 3: The model performance of ionic 2D low-profile PSM and 3D PSM. a, The physical image of ionic 2D low-profile PSM proposed inwang2023passively. b, The error map between model predictions and the ground truth control input vectors of ionic 2D low-profile PSM. c, The error map of ionic 2D low-profile PSM showcasing the point cloud of reproduced shapes with the ground truth point cloud. d, The comparison of R2 score across various training models for ionic 2D low-profile PSM. e, The exploded image of ionic 3D PSM assembled by 6 pieces of ionic 2D low-profile PSM. f, The unfolded error map between model predictions and the ground truth control input vectors of ionic 3D PSM. g, The 3D error map of ionic 2D low-profile PSM showcasing the point cloud of reproduced shapes with the ground truth point cloud. There are two angles of view to show the entire 6 surfaces of the cube. h, The comparison of R2 score across various training models for ionic 3D PSM.
  • Figure 4: The model performance of thermal 3D PSM and pneumatic 3D PSM and the comparison between the model performance of SMNet with KPConv and PointNet++. a, Expanding from ionic 3D PSM based on the bending principle to thermal 3D PSM based on volume change. b, Expanding from ionic 3D PSM based on the bending principle to pneumatic 3D PSM based on surface buckling. c & e, the unfolded error map between model predictions and the ground truth control input vectors of thermal 3D PSM and pneumatic 3D PSM, respectively. d & f, the 3D error map of thermal 3D PSM and pneumatic 3D PSM showcasing the point cloud of reproduced shapes with the ground truth point cloud. There are two angles of view to show the entire 6 surfaces of the cube. f, the model performance comparison between KPConv, PointNet++, and SMNet. We laid out the error maps of the reproduced point cloud and the ground-truth point cloud into six faces, arranged in two rows. Additionally, we compared each dimension of the predicted input vector with the ground-truth input vector, and linearly displayed the error of each dimension below the point cloud error maps.
  • Figure 5: The demonstration of SMNet on inverse control of 3D shape morphing devices. a, The detailed procedures of the demonstrations. b, The reproduced point cloud of 3 different mechanisms. 'Demo 1' and 'Demo 2' present two shapes formed by manually molding the clay. 'Demo 3' is made by software with high surface complexity. c, The similarity between the reproduced point cloud with the target point cloud by using Chamfer Distance (CD), standard deviation of distance, and Hausdorff Distance (HD). All of the data has been normalized to 1. To facilitate a better comparison among the other cases, the bars for pneumatic actuators employ a truncated axis.
  • ...and 2 more figures