Grounded Task Axes: Zero-Shot Semantic Skill Generalization via Task-Axis Controllers and Visual Foundation Models

TL;DR

Grounded Task-Axes address zero-shot generalization of robotic manipulation by grounding modular task-axis controllers to object keypoints using visual foundation models. The GTA framework decomposes skills into prioritized controllers and grounds them in new scenes through example keypoints matched by SD-DINO, complemented by a feature map that blends semantic and geometric cues. The approach enables zero-shot transfer across diverse objects for tasks such as pan scraping, pouring, and screwing without demonstrations or retraining, with high positional and angular accuracy. This modular, semantic grounding strategy supports scalable skill reuse in open-world robotics and is robust to appearance and geometry variation.

Abstract

Transferring skills between different objects remains one of the core challenges of open-world robot manipulation. Generalization needs to take into account the high-level structural differences between distinct objects while still maintaining similar low-level interaction control. In this paper, we propose an example-based zero-shot approach to skill transfer. Rather than treating skills as atomic, we decompose skills into a prioritized list of grounded task-axis (GTA) controllers. Each GTAC defines an adaptable controller, such as a position or force controller, along an axis. Importantly, the GTACs are grounded in object key points and axes, e.g., the relative position of a screw head or the axis of its shaft. Zero-shot transfer is thus achieved by finding semantically-similar grounding features on novel target objects. We achieve this example-based grounding of the skills through the use of foundation models, such as SD-DINO, that can detect semantically similar keypoints of objects. We evaluate our framework on real-robot experiments, including screwing, pouring, and spatula scraping tasks, and demonstrate robust and versatile controller transfer for each.

Figures (8)

• Figure 1: Grounded Task-Axes (GTAs): Our system generalizes task-axis controllers zero-shot via vision-based keypoint correspondences.
• Figure 2: Vision pipeline for grounding task-axes: Given a reference object annotated with keypoints and a target object, GTA uses SD-DINO as a vision backbone to extract visual features. We perform keypoint matching to find corresponding keypoints on the target object. Using these keypoints and 3D point cloud data, we ground the object-centered task-axis controllers in real-world.
• Figure 3: (a) Example Keypoints and Task-Axes for a spatula object. (b) Zero-shot generalization to any object configuration.
• Figure 4: Visualization of cosine similarity maps on target object image according to different pixel keypoints selected on the reference object.
• Figure 5: (Left) Target tasks: Scraping a pan, pouring, and screwing. (Middle) Example object configurations with keypoint annotations from our in-lab data collection (Right) Grounded Task-Axes derived from keypoints.