Proactive tactile exploration for object-agnostic shape reconstruction from minimal visual priors

TL;DR

The paper addresses robust 3D shape reconstruction for grasping under limited visual priors by combining a two-step, template-guided global fit with local deformation, and an active tactile exploration strategy that minimizes contacts while reducing the risk of contact failure. The method uses an ellipsoid primitive as a global prior and refines the surface via thin-plate-spline-based local displacement, with uncertainty propagated from attractors to mesh vertices. An active exploration criterion balances exploration efficiency and safety, selecting tactile contacts to reduce total surface uncertainty without risking futile contacts. Demonstrations in simulation and on a real robotic setup show fast convergence with few tactile measurements and fewer contact failures, highlighting practical viability for visuo-tactile object reconstruction in manipulation tasks.

Abstract

The perception of an object's surface is important for robotic applications enabling robust object manipulation. The level of accuracy in such a representation affects the outcome of the action planning, especially during tasks that require physical contact, e.g. grasping. In this paper, we propose a novel iterative method for 3D shape reconstruction consisting of two steps. At first, a mesh is fitted on data points acquired from the object's surface, based on a single primitive template. Subsequently, the mesh is properly adjusted to adequately represent local deformities. Moreover, a novel proactive tactile exploration strategy aims at minimizing the total uncertainty with the least number of contacts, while reducing the risk of contact failure in case the estimated surface differs significantly from the real one. The performance of the methodology is evaluated both in 3D simulation and on a real setup.

Figures (6)

• Figure 1: Overview of the proposed pipeline. To ease the comprehension of the methodology, the shape reconstruction of a 2d bottle-shape object is considered, where a path graph is used, instead of a 3d mesh. The colored blocks correspond to each of the individual algorithmic components taking part in the reconstruction (details in Section \ref{['sec:meth']}). In each block, the top instance illustrates the outcome of the process during the first iteration where only the minimal visual prior (light blue) is available, while the bottom illustration concerns the corresponding outcome when five tactile surface points (dark blue) have been collected.
• Figure 2: Top left: The tactile sensor is a variant of the Shokac Chip 6DoF-P18 by Touchence touchsense_shokac and measures the force/torque applied along all three axes. Bottom left: At each iteration, the robot moves within a predefined distance $d$ before (red) and after (blue) the candidate contact point (green) along its estimated normal vector. In case of no contact, the attempt is considered as failure. Rest with "X": Other cases of contact failure if the sensing part (black surface) does not directly touch the object. Right: The real experimental setup.
• Figure 3: First three iterations of the proposed active tactile exploration strategy (Section \ref{['sec:pselect']}). At the side of the visual points (cyan), the reconstructed mesh (red-green) fits the actual one (grey). At the other side tactile exploration is required. Following the proposed algorithm, a path of tactile points (blue) is created from a neighborhood close to the visual data up to the unexplored side of the surface, resembling a depth first search.
• Figure 4: Iterative progression of the proposed methodology. The first two rows corresponds to the evaluation performed in simulation, while the last one took place on the real experimental setup (Sec. \ref{['sec:real_exp']}). From left to right: target mesh, 6 instances of the reconstruction at 0 (initial), 1, 2, 5, 10 and 30 iterations and the resulting mesh after 30 iterations. The intermediate instances depict the predicted surface overlapped on the target shape, where green and red regions denote low and high uncertainty, respectively. Visual points are depicted with cyan color, while tactile points are visualized as blue points (zoom is required).
• Figure 5: Reconstruction of several objects (from left to right: coffee can, box of sugar, mini soccer ball, pear, mustard container) from YCB dataset, after 50 iterations of the methodology (including contact failure iterations). Top: Real mesh of the object and collected attractors (visual in cyan, tactile in blue). Bottom: Reconstructed shape colored with uncertainty (high in red, low in green).