PokeNet: Learning Kinematic Models of Articulated Objects from Human Observations

TL;DR

PokeNet tackles articulation learning for multi-DoF objects from a single human demonstration without prior object knowledge. It adopts a DETR-style set-prediction architecture over a sequence of point clouds to output a fixed set of joints, each with type, axis, anchor, and an inferred manipulation order, plus per-frame joint states; permutation-invariant Hungarian matching enables flexible joint counts. The model is trained with a composite loss that combines confidence, type, axis, position, order, and state terms and evaluated on large simulated and real-world articulated-object datasets, showing substantial improvements over state-of-the-art baselines and generalization to unseen categories and scales. A real-robot demonstration shows the predicted articulation parameters enable feasible manipulation trajectories, and a large annotated real-world dataset of articulated objects is released. This work advances safe, data-efficient articulation understanding for manipulation, enabling robots to operate with unseen articulated objects without requiring extensive priors.

Abstract

Articulation modeling enables robots to learn joint parameters of articulated objects for effective manipulation which can then be used downstream for skill learning or planning. Existing approaches often rely on prior knowledge about the objects, such as the number or type of joints. Some of these approaches also fail to recover occluded joints that are only revealed during interaction. Others require large numbers of multi-view images for every object, which is impractical in real-world settings. Furthermore, prior works neglect the order of manipulations, which is essential for many multi-DoF objects where one joint must be operated before another, such as a dishwasher. We introduce PokeNet, an end-to-end framework that estimates articulation models from a single human demonstration without prior object knowledge. Given a sequence of point cloud observations of a human manipulating an unknown object, PokeNet predicts joint parameters, infers manipulation order, and tracks joint states over time. PokeNet outperforms existing state-of-the-art methods, improving joint axis and state estimation accuracy by an average of over 27% across diverse objects, including novel and unseen categories. We demonstrate these gains in both simulation and real-world environments.

Figures (4)

• Figure 1: We propose a novel framework that learns the joint parameters and manipulation order of articulated objects directly from human demonstrations. Leveraging human interactions allows our method to reason about occluded joints, while requiring no prior object knowledge. It generalizes to unseen object categories and achieves state-of-the-art performance in both simulation and real-world settings.
• Figure 2: Overview of PokeNet. Our model takes a sequence of point clouds as input. Each frame is encoded with PointNet++ to extract spatial features, and [CLS] tokens are passed through a transformer encoder to capture temporal dependencies. A DETR style joint decoder with learnable queries attends to these temporal features to predict an ordered kinematic slot model, including joint type, axis direction, anchor point, confidence, and manipulation order. An auxiliary decoder additionally predicts per-frame joint states.
• Figure 3: This figure shows Sawyer robot manipulating the two joints of the fridge in the order of demonstration estimated by Pokenet. (a) and (b) shows human demonstrations while (c) and (d) shows robot manipulating the object.
• Figure 4: Comparison of PokeNet and GAPartNet on different object categories. Left: axis direction accuracy; Right: axis displacement accuracy (higher is better). GAPartNet struggles with fully shut parts, while PokeNet uses human demonstrations to generalize robustly.