Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PokeFlex: Towards a Real-World Dataset of Deformable Objects for Robotic Manipulation

Jan Obrist, Miguel Zamora, Hehui Zheng, Juan Zarate, Robert K. Katzschmann, Stelian Coros

TL;DR

The paper presents PokeFlex, a real-world dataset of deformable objects captured through volumetric 3D reconstruction while a robot performs poking manipulations, addressing the data scarcity barrier in deformable-object robotics. It provides 5 objects with rich per-frame data (3D meshes, forces/torques, end-effector poses, and camera records) and demonstrates online 3D mesh deformation prediction from a single image using a Real-NVP model at ~125 Hz. The work establishes a foundation for data-driven manipulation, material parameter estimation, and online mesh-reconstruction methods, with plans to expand to 3D-printed objects and broader manipulation strategies to enhance diversity and reproducibility. Overall, PokeFlex enables practical progress toward real-world manipulation of deformable objects and sets the stage for downstream tasks like policy learning and material identification in robotics.

Abstract

Advancing robotic manipulation of deformable objects can enable automation of repetitive tasks across multiple industries, from food processing to textiles and healthcare. Yet robots struggle with the high dimensionality of deformable objects and their complex dynamics. While data-driven methods have shown potential for solving manipulation tasks, their application in the domain of deformable objects has been constrained by the lack of data. To address this, we propose PokeFlex, a pilot dataset featuring real-world 3D mesh data of actively deformed objects, together with the corresponding forces and torques applied by a robotic arm, using a simple poking strategy. Deformations are captured with a professional volumetric capture system that allows for complete 360-degree reconstruction. The PokeFlex dataset consists of five deformable objects with varying stiffness and shapes. Additionally, we leverage the PokeFlex dataset to train a vision model for online 3D mesh reconstruction from a single image and a template mesh. We refer readers to the supplementary material and to our website ( https://pokeflex-dataset.github.io/ ) for demos and examples of our dataset.

PokeFlex: Towards a Real-World Dataset of Deformable Objects for Robotic Manipulation

TL;DR

The paper presents PokeFlex, a real-world dataset of deformable objects captured through volumetric 3D reconstruction while a robot performs poking manipulations, addressing the data scarcity barrier in deformable-object robotics. It provides 5 objects with rich per-frame data (3D meshes, forces/torques, end-effector poses, and camera records) and demonstrates online 3D mesh deformation prediction from a single image using a Real-NVP model at ~125 Hz. The work establishes a foundation for data-driven manipulation, material parameter estimation, and online mesh-reconstruction methods, with plans to expand to 3D-printed objects and broader manipulation strategies to enhance diversity and reproducibility. Overall, PokeFlex enables practical progress toward real-world manipulation of deformable objects and sets the stage for downstream tasks like policy learning and material identification in robotics.

Abstract

Advancing robotic manipulation of deformable objects can enable automation of repetitive tasks across multiple industries, from food processing to textiles and healthcare. Yet robots struggle with the high dimensionality of deformable objects and their complex dynamics. While data-driven methods have shown potential for solving manipulation tasks, their application in the domain of deformable objects has been constrained by the lack of data. To address this, we propose PokeFlex, a pilot dataset featuring real-world 3D mesh data of actively deformed objects, together with the corresponding forces and torques applied by a robotic arm, using a simple poking strategy. Deformations are captured with a professional volumetric capture system that allows for complete 360-degree reconstruction. The PokeFlex dataset consists of five deformable objects with varying stiffness and shapes. Additionally, we leverage the PokeFlex dataset to train a vision model for online 3D mesh reconstruction from a single image and a template mesh. We refer readers to the supplementary material and to our website ( https://pokeflex-dataset.github.io/ ) for demos and examples of our dataset.
Paper Structure (3 sections, 4 figures)

This paper contains 3 sections, 4 figures.

Table of Contents

  1. INTRODUCTION
  2. RESULTS
  3. CONCLUSIONS AND FUTURE WORK

Figures (4)

  • Figure 1: Setup for recording PokeFlex dataset: Husky dual arm robot is placed in a volumetric capture system within reach of the object. While the husky dual arm robot pokes the object, the deformations are recorded from the capture system at 30 fps. The robot records the position of the end effector and its acting forces and torques at 100 Hz.
  • Figure 2: Two step mesh postprocessing: (a)$\rightarrow$(b) Using vertical and horizontal plane for clipping to remove most parts of the robotic system. (b)$\rightarrow$(c) Using frame based mesh clipping with 3D end effector model as mask.
  • Figure 3: Examples of reconstructed 3D meshes with and without texture (top and bottom respectively) for toilet paper roll (left) and firm pillow (right).
  • Figure 4: Example prediction for the toilet paper roll object: Template mesh (left), predicted deformation (middle) and ground truth deformation (right).