Online Estimation and Manipulation of Articulated Objects

TL;DR

The paper tackles the challenge of online estimation and manipulation of unknown articulated objects by fusing learned visual priors with proprioceptive sensing in a factor-graph framework grounded in Screw Theory. It introduces an uncertainty-aware articulation prediction network, a new per-point affordance factor, and a force-based factor to drive online updates during interaction, enabling robust closed-loop manipulation. The approach is implemented in a three-module system with Initialization, Estimation, and Motion Generation, and validated in both simulation and real hardware, achieving 75% autonomous opening on unseen objects and 15/20 success in shared-autonomy experiments. This work advances multi-modal perception for articulated object manipulation and provides a practical path toward robust service-robot interaction in home environments.

Abstract

From refrigerators to kitchen drawers, humans interact with articulated objects effortlessly every day while completing household chores. For automating these tasks, service robots must be capable of manipulating arbitrary articulated objects. Recent deep learning methods have been shown to predict valuable priors on the affordance of articulated objects from vision. In contrast, many other works estimate object articulations by observing the articulation motion, but this requires the robot to already be capable of manipulating the object. In this article, we propose a novel approach combining these methods by using a factor graph for online estimation of articulation which fuses learned visual priors and proprioceptive sensing during interaction into an analytical model of articulation based on Screw Theory. With our method, a robotic system makes an initial prediction of articulation from vision before touching the object, and then quickly updates the estimate from kinematic and force sensing during manipulation. We evaluate our method extensively in both simulations and real-world robotic manipulation experiments. We demonstrate several closed-loop estimation and manipulation experiments in which the robot was capable of opening previously unseen drawers. In real hardware experiments, the robot achieved a 75% success rate for autonomous opening of unknown articulated objects.

Figures (11)

• Figure 1: Top row: a cabinet with a set of visually identical doors. Their different articulations are only revealed once open. It would not be possible from visual inspection alone to predict how each door opens. Middle and Bottom rows: the robot autonomously opens each of the cabinet doors while estimating articulation online.
• Figure 2: The factor graph shows the variables we are estimating: $\mathbf{T_{\mathtt{A}}}(t), \mathbf{T_{\mathtt{B}}}(t), \theta(t)$ and $\xi$, which exists at only one time step in the factor graph. We show three time steps, including the initial visual affordance factor, which provides a prior estimate on $\xi$ as a unary factor.
• Figure 3: Example affordance predictions from the neural network from Buchanan2024: prismatic left and revolute right. The small red lines are the output of the network, predicting articulation flow on the segmented points. The large red and yellow arrows indicated the resulting joint prediction from plane fitting as was done in Buchanan2024.
• Figure 4: Example output of articulation flow prediction with covariances. Left: rendering of simulated sliding door, note the axis. Center left: green point cloud measurement of the door with the ground truth articulation flow shown as red lines. Center right: predicted articulation flow shown as red lines. The neural network has mistaken the door for revolute with a joint on the right side of the door frame. Right: the covariance for each articulation flow vector. There is the highest covariance in the x direction, showing a high degree of uncertainty with this articulation. The next highest uncertainty is in the y direction, followed by z.
• Figure 5: Example of relationship between applied direction of motion (grey), measured reaction force, and valid direction of motion. Top: a downward force is applied to a prismatic joint, which results in the upward reaction force $\hat{\mathbf{F}}$. This is orthonormal to a plane (gray dotted plane) on which we know the valid motion $\mathbf{v}_{valid}$ must lie. Bottom: a force is applied towards the hinge of a revolute joint, resulting in a reaction force perpendicular to the direction of motion and orthonormal to a plane on which $\mathbf{v}_{valid}$ lies.