Adaptive Manipulation Potential and Haptic Estimation for Tool-Mediated Interaction

TL;DR

A parameterized Equilibrium Manifold is introduced as a unified representation for tool-mediated interaction, and a closed-loop framework that integrates haptic estimation, online planning, and adaptive stiffness control is developed.

Abstract

Achieving human-level dexterity in contact-rich, tool-mediated manipulation remains a significant challenge due to visual occlusion and the underdetermined nature of haptic sensing. This paper introduces a parameterized Equilibrium Manifold (EM) as a unified representation for tool-mediated interaction, and develops a closed-loop framework that integrates haptic estimation, online planning, and adaptive stiffness control. We establish a physical-geometric duality using an adaptive manipulation potential incorporating a differentiable contact model, which induces the manifold's geometric structure and ensures that complex physical interactions are encapsulated as continuous operations on the EM. Within this framework, we reformulate haptic estimation as a manifold parameter estimation problem. Specifically, a hybrid inference strategy (haptic SLAM) is employed in which discrete object shapes are classified via particle filtering, while the continuous object pose is estimated using analytical gradients for efficient optimization. By continuously updating the parameters of the manipulation potential, the framework dynamically reshapes the induced EM to guide online trajectory replanning and implement uncertainty-aware impedance control, thereby closing the perception-action loop. The system is validated through simulation and over 260 real-world screw-loosening trials. Experimental results demonstrate robust identification and manipulation success in standard scenarios while maintaining accurate tracking. Furthermore, ablation studies confirm that haptic SLAM and uncertainty-aware stiffness modulation outperform fixed impedance baselines, effectively preventing jamming during tight tolerance interactions.

Figures (22)

• Figure 1: Tool-mediated haptic manipulation under visual occlusion. (A) Humans can perform skillful haptic manipulation with tools despite unreliable visual information, although sensing is limited to net forces and torques at the human–tool interface. (B) Multiple contact points may exist at the tool tip while only a single net wrench is observed, leading to contact ambiguity. (C) Humans rely on cognitive reasoning and internal models of tool use to infer task-relevant object properties, such as pose and geometry, through haptic interaction.
• Figure 2: Overall workflow of tool-mediated manipulation framework. Left: the robot uses a tool to interact with the object and senses the resulting net wrench $\overline{\bm{\mathcal{F}}}$. Middle: $\overline{\bm{\mathcal{F}}}$ is fed into a haptic state estimator that incorporates a manifold-based internal physical model to predict the expected wrench $\bm{\mathcal{F}}$. The discrepancy between the observed and predicted wrenches—termed haptic mismatch—is used to update the estimated object pose ${\boldsymbol{{\theta}}}$ and type $s$. Right: Based on the optimal estimates $(s^*, {\boldsymbol{{\theta}}}^*)$ and associated uncertainty ${\boldsymbol{{\Sigma}}}_{{\boldsymbol{{\theta}}}}$, a planner and controller generate the control ${\boldsymbol{{u}}}(t)$ and stiffness ${\boldsymbol{{K}}}_c$ for robot.
• Figure 3: Parameterized equilibrium manifolds and induced motion. For fixed environment parameters $(s,{\boldsymbol{{\theta}}})$, the quasi-static interaction between the robot and the environment defines an equilibrium manifold $\mathcal{M}_{eq}^{(s,{\boldsymbol{{\theta}}})}$ embedded in the state-control space. Different values of the discrete parameter $s$ select different manifolds, while the continuous parameter ${\boldsymbol{{\theta}}}$ continuously deforms the manifold geometry within a given structural model. For fixed $(s,{\boldsymbol{{\theta}}})$, control inputs ${\boldsymbol{{u}}}$ induce equilibrium configurations ${\boldsymbol{{z}}}^*({\boldsymbol{{u}}};{\boldsymbol{{\theta}}},s)$ along the manifold. The resulting motion on $\mathcal{M}_{eq}^{(s,{\boldsymbol{{\theta}}})}$ is governed by an adaptive ODE.
• Figure 4: Instantiation of the manipulation potential $W^{(s)}({\boldsymbol{{z}}},{\boldsymbol{{u}}};{\boldsymbol{{\theta}}})$ through contact geometry and impedance control. Left: Schematic representation of the general case. Right: Illustrative example of a spanner–screw operation.
• Figure 5: Workflow of the hybrid inference in Haptic SLAM. For each discrete shape hypothesis $s^{(i)}$, multiple pose particles ${\boldsymbol{{\theta}}}^{(j)}[s^{(i)}]$ are initialized to represent candidate configurations. These particles process sensed haptic observations $\overline{\mathcal{F}}$ in parallel, independently updating their pose parameters ${\boldsymbol{{\theta}}}^{(j)}[s^{(i)}]$ via a local optimizer driven by the internal physical model. A representative pose ${\boldsymbol{{\theta}}}^*[s^{(i)}]$ is then identified for each shape, followed by a MAP selection and clustering process to determine the estimated object shape $s^*$ and pose ${\boldsymbol{{\theta}}}^*$ for the current batch. This inference cycle is re-executed in each subsequent batch as new haptic observations become available.