Learning Sequential Kinematic Models from Demonstrations for Multi-Jointed Articulated Objects

TL;DR

The paper tackles generalizable manipulation of multi-DoF articulated objects by learning representations from human demonstrations. It introduces Object Kinematic Sequence Machines (OKSMs) to encode joint types, parameters, and manipulation order, and the Pokenet network to estimate OKSMs from sequences of point clouds, feeding a motion planner for manipulation. Extensive simulated and real-world datasets show that Pokenet improves joint-axis and state estimation by over 20–30% compared with baselines and enables real-world robot manipulation on a Sawyer using inverse kinematics-based planning. This work advances category-agnostic articulation understanding and practical planning-based manipulation in unstructured environments.

Abstract

As robots become more generalized and deployed in diverse environments, they must interact with complex objects, many with multiple independent joints or degrees of freedom (DoF) requiring precise control. A common strategy is object modeling, where compact state-space models are learned from real-world observations and paired with classical planning. However, existing methods often rely on prior knowledge or focus on single-DoF objects, limiting their applicability. They also fail to handle occluded joints and ignore the manipulation sequences needed to access them. We address this by learning object models from human demonstrations. We introduce Object Kinematic Sequence Machines (OKSMs), a novel representation capturing both kinematic constraints and manipulation order for multi-DoF objects. To estimate these models from point cloud data, we present Pokenet, a deep neural network trained on human demonstrations. We validate our approach on 8,000 simulated and 1,600 real-world annotated samples. Pokenet improves joint axis and state estimation by over 20 percent on real-world data compared to prior methods. Finally, we demonstrate OKSMs on a Sawyer robot using inverse kinematics-based planning to manipulate multi-DoF objects.

Figures (4)

• Figure 1: Figure illustrating our framework enabling the robot to manipulate a microwave from a single human demonstration. We first capture the human demonstration as a sequence of point clouds. Pokenet takes this sequence and predicts an OKSM for the object. The motion planner then generates a manipulation plan for the object using the OKSM's sequence ordering and object parameterization as predicted by our model along a specified direction, such as "open" and a given grasp point.
• Figure 2: This figure shows the robot manipulating four real-world test objects using OKSMs predicted by Pokenet. (a) Microwave with a single revolute joint. (b) Drawer with a prismatic joint. (c) Fridge with two revolute and one prismatic joint. (d) Dishwasher with one revolute and one prismatic joint.
• Figure 3: This figure shows Sawyer robot manipulating the two joints of the fridge in the order of demonstration estimated by Pokenet. (a) and (b) shows human demonstrations while (c) and (d) shows robot manipulating the object.
• Figure 4: Our model processes a sequence of point clouds, each encoded by PointNet to extract spatial features. These features are stacked and passed through a transformer encoder to capture temporal information. The encoder output is averaged and fed to an MLP to predict the OKSM.