A Unified Framework for Probabilistic Dynamic-, Trajectory- and Vision-based Virtual Fixtures

TL;DR

The paper addresses the need for flexible automation in manipulation tasks by introducing a unified probabilistic VF framework that merges coarse DS-based guidance, precise position-based trajectory fixtures, and very-precise visual servoing, all within a geometry-aware, manifold-informed arbitration scheme. It advances variable impedance control by deriving fully populated stiffness matrices from learned covariances, enabling translation-rotation couplings and cross-manifold applicability. The approach supports fully manual, semi-automated, and autonomous operation and demonstrates robust fusion across multiple fixtures and manifolds on diverse robotic platforms, including space-grade systems, with initial user evaluations indicating high usability. Overall, the framework offers a principled, uncertainty-aware path to multi-fixture automation that can adapt to task phase, input modality, and geometry, paving the way for broader deployment in industrial and space contexts.

Abstract

Probabilistic Virtual Fixtures (VFs) enable the adaptive selection of the most suitable haptic feedback for each phase of a task, based on learned or perceived uncertainty. While keeping the human in the loop remains essential, for instance, to ensure high precision, partial automation of certain task phases is critical for productivity. We present a unified framework for probabilistic VFs that seamlessly switches between manual fixtures, semi-automated fixtures (with the human handling precise tasks), and full autonomy. We introduce a novel probabilistic Dynamical System-based VF for coarse guidance, enabling the robot to autonomously complete certain task phases while keeping the human operator in the loop. For tasks requiring precise guidance, we extend probabilistic position-based trajectory fixtures with automation allowing for seamless human interaction as well as geometry-awareness and optimal impedance gains. For manual tasks requiring very precise guidance, we also extend visual servoing fixtures with the same geometry-awareness and impedance behavior. We validate our approach experimentally on different robots, showcasing multiple operation modes and the ease of programming fixtures.

Figures (22)

• Figure 1: Overview of our unified framework. We propose a new type of learned, ds based vf to assist an operator with coarse guidance in progressing along the task while staying near the training data. For precise guidance, position-based trajectory fixtures support the human operator while geometric visual servoing fixtures provide very precise guidance. Core of our framework is a novel variable impedance control scheme as well as an optimal arbitration of all probabilistic fixture wrenches.
• Figure 2: Manifolds used in this work inspired by and using the notation of ti2023geometric. Depending on the task, properties can be expressed more efficiently in cylindrical ($\mathcal{M}_2$) or spherical ($\mathcal{M}_3$) compared to Cartesian ($\mathcal{M}_1$) coordinates. The coordinate systems in each image depict the orientation basis, i.e. the unit quaternion $(0, 0, 0, 1)^\top$ for different positions on the manifold.
• Figure 3: The coordinate system defined with respect to the robot allows to account for different object placements in the workspace while the specification of coordinate systems relative to the coordinate system is crucial for cylindrical and spherical coordinates. Finally, the frame depends on the current end effector pose of the robot.
• Figure 4: 2D motion policy using a . Black arrows visualize velocities in the training data calinon2017learning. The output is shown on the left (\ref{['sec:non_parametric_DS']}), blue ellipsoids and red arrows depict centers, position covariances and velocities of the Gaussians in the reference . The stabilizing policy (\ref{['sec:dynamic_fixtures:base_policy']}) is shown in the middle, final velocities resulting from the arbitration on the right. The colormap of each plot depicts $\mathrm{log}(\mathrm{det}(\bm{\Sigma}))$.
• Figure 5: A torque-controlled 7- manipulator (left) is used in hand-guided mode for evaluating individual fixtures in \ref{['sec:evaluation:coupled_stiffness', 'sec:evaluation:position_stiffness', 'sec:evaluation:dynamic_vf']}, as well as their combinations in \ref{['sec:evaluation:dynamic_vs_hug', 'sec:evaluation:all_fixtures_sara']}. A dual arm setup with two torque-controlled 7- manipulators (center) is used for evaluating the fixtures in a teleoperation task in \ref{['sec:evaluation:dynamic_vs_hug']}. A space-ready, 4- robot arm (right) is used with fully automated fixtures (\ref{['sec:evaluation:dynamic_vf', 'sec:evaluation:auto_pb_robograv', 'sec:evaluation:dynamic_pb_robograv']}.)