Soft Contact Simulation and Manipulation Learning of Deformable Objects with Vision-based Tactile Sensor

TL;DR

This work introduces a deformable object manipulation framework that combines a soft contact simulator capable of elastic, plastic, and elastoplastic deformation with vision-based tactile observations to train transferable policies. By using VBTS as the end-effector and MLS-MPM for deformable dynamics, the authors build a simulation-to-real bridge that enables RL policies (via TD3 and expert demonstrations) to control deformable objects such as cylinder- and sphere-shaped plasticine on real hardware with 90% success in challenging tasks. The approach yields transferable observations—relative position, squeezed area, and object contour—that support sim-to-real transfer and robust manipulation across varying material properties. The study demonstrates both the efficacy of the simulation framework and the practical viability of sim-to-real transfer for deformable object manipulation in robotics.

Abstract

Deformable object manipulation is a classical and challenging research area in robotics. Compared with rigid object manipulation, this problem is more complex due to the deformation properties including elastic, plastic, and elastoplastic deformation. In this paper, we describe a new deformable object manipulation method including soft contact simulation, manipulation learning, and sim-to-real transfer. We propose a novel approach utilizing Vision-Based Tactile Sensors (VBTSs) as the end-effector in simulation to produce observations like relative position, squeezed area, and object contour, which are transferable to real robots. For a more realistic contact simulation, a new simulation environment including elastic, plastic, and elastoplastic deformations is created. We utilize RL strategies to train agents in the simulation, and expert demonstrations are applied for challenging tasks. Finally, we build a real experimental platform to complete the sim-to-real transfer and achieve a 90% success rate on difficult tasks such as cylinder and sphere. To test the robustness of our method, we use plasticine of different hardness and sizes to repeat the tasks including cylinder and sphere. The experimental results show superior performances of deformable object manipulation with the proposed method.

Figures (10)

• Figure 1: System overview. In the simulation, the contact simulation provides the observations of relative position, squeezed area, and object contour for the training of the agent. The agent controls the robot to move and update the observation until training is finished. In reality, the VBTS provides the observations for the agent trained in the simulation. In the end, the agent controls the robot to move under the desired motion and completes the sim-to-real.
• Figure 2: (A): Moving least squares material point method diagram. Blue particles represent gel layers, and red particles represent deformable objects. In this diagram, a gel layer is pressed against the deformable object, and the object is distorted. (B): Observations. The observation named relative position contains the middle point of the gel layer and deformable object. The observation named squeezed area contains the deformation area of the gel layer. The observation named object contour contains the contact and non-contact areas of the object.
• Figure 3: (A): The principle of the total internal reflection (TIR). (B): The 3D model of the vision-based tactile sensor (VBTS). The real-world model of the VBTS and the result of the TIR are also shown in this Figure.
• Figure 4: The real tactile image and segmented results of the observations in reality. The red object is plasticine, and we use VBTS to press it to introduce observations in reality.
• Figure 5: Task diagrams. These diagrams show the training results. In position control, the deformable object is desired to move to a specific position. In squeeze, the deformable object is squeezed to a desired thickness. In the task cylinder and sphere, the deformable object is kneaded into a cylinder and sphere, respectively. These tasks are of increasing difficulty.