ArtReg: Visuo-Tactile based Pose Tracking and Manipulation of Unseen Articulated Objects

TL;DR

This paper tackles the challenge of tracking and manipulating unseen articulated objects without prior geometric or kinematic knowledge. It introduces ArtReg, a SE(3) Lie-group Unscented Kalman Filter that fuses visuo-tactile data for robust 6-DoF pose tracking, coupled with interactive perception to detect articulation and a closed-loop visuo-tactile controller for goal-driven manipulation. The framework is validated through extensive real-robot experiments with two robots, demonstrating reliable articulation detection, accurate pose tracking (often sub-centimeter to a few centimeters), and successful manipulation under challenging lighting and backgrounds. Benchmarking on the PartNet-Mobility dataset indicates significant performance gains over state-of-the-art baselines, underscoring the practical impact of combining tactile sensing with visual perception for autonomous interaction with unseen articulated objects.

Abstract

Robots operating in real-world environments frequently encounter unknown objects with complex structures and articulated components, such as doors, drawers, cabinets, and tools. The ability to perceive, track, and manipulate these objects without prior knowledge of their geometry or kinematic properties remains a fundamental challenge in robotics. In this work, we present a novel method for visuo-tactile-based tracking of unseen objects (single, multiple, or articulated) during robotic interaction without assuming any prior knowledge regarding object shape or dynamics. Our novel pose tracking approach termed ArtReg (stands for Articulated Registration) integrates visuo-tactile point clouds in an unscented Kalman Filter formulation in the SE(3) Lie Group for point cloud registration. ArtReg is used to detect possible articulated joints in objects using purposeful manipulation maneuvers such as pushing or hold-pulling with a two-robot team. Furthermore, we leverage ArtReg to develop a closed-loop controller for goal-driven manipulation of articulated objects to move the object into the desired pose configuration. We have extensively evaluated our approach on various types of unknown objects through real robot experiments. We also demonstrate the robustness of our method by evaluating objects with varying center of mass, low-light conditions, and with challenging visual backgrounds. Furthermore, we benchmarked our approach on a standard dataset of articulated objects and demonstrated improved performance in terms of pose accuracy compared to state-of-the-art methods. Our experiments indicate that robust and accurate pose tracking leveraging visuo-tactile information enables robots to perceive and interact with unseen complex articulated objects (with revolute or prismatic joints).

Figures (18)

• Figure 1: Experimental setup: A Franka Emika Panda robot with a Azure Kinect DK RGB-D vision sensor and a Universal Robots UR5 sensorised with tactile sensor arrays on the Robotiq Gripper, with unknown articulated objects in the workspace. The robots perform interactive perception to detect possible articulation structure in the objects. The objects are tracked using our ArtReg algorithm and the 6 degree-of-freedom (DoF) tracking information is used for goal-driven manipulation.
• Figure 2: Our proposed framework: (a) Visuo-tactile-based pose tracking method termed ArtReg, (b) Interactive visuo-tactile perception for articulation detection, and (c) Visuo-tactile-based closed-loop control for goal-driven manipulation.
• Figure 3: (a) The experimental setup shown along with the description of the state and measurement vector. (b) Visualization of the basic manifold operations: exponential map ($\varphi(\cdot)$) and logarithmic map ($\varphi^{-1}(\cdot)$). (c) Our ArtReg algorithm which is a manifold unscented Kalman filter visualised with operations on the state manifold $\mathcal{M}$ and measurement manifold $\mathcal{M}_{obs}$.
• Figure 4: Interactive perception for articulation detection: (a) push action for revolute joints, (b) grasp and pull action for prismatic joints.
• Figure 5: Goal-driven closed loop control system