ConceptACT: Episode-Level Concepts for Sample-Efficient Robotic Imitation Learning

TL;DR

ConceptACT addresses the inefficiency of purely low-level imitation learning by letting human demonstrators annotate episodes with high-level semantic concepts. It integrates these episode-level concepts through a class-aware Concept Transformer within ACT, aligning attention with concepts during training while keeping deployment agnostic to semantics. Empirically, ConceptACT-Transformer delivers faster convergence and stronger sample efficiency than standard ACT and language-conditioned baselines, especially in concept-rich tasks. This architectural grounding of semantic supervision provides a practical, scalable path to more data-efficient robotic imitation learning with real-world deployment benefits.

Abstract

Imitation learning enables robots to acquire complex manipulation skills from human demonstrations, but current methods rely solely on low-level sensorimotor data while ignoring the rich semantic knowledge humans naturally possess about tasks. We present ConceptACT, an extension of Action Chunking with Transformers that leverages episode-level semantic concept annotations during training to improve learning efficiency. Unlike language-conditioned approaches that require semantic input at deployment, ConceptACT uses human-provided concepts (object properties, spatial relationships, task constraints) exclusively during demonstration collection, adding minimal annotation burden. We integrate concepts using a modified transformer architecture in which the final encoder layer implements concept-aware cross-attention, supervised to align with human annotations. Through experiments on two robotic manipulation tasks with logical constraints, we demonstrate that ConceptACT converges faster and achieves superior sample efficiency compared to standard ACT. Crucially, we show that architectural integration through attention mechanisms significantly outperforms naive auxiliary prediction losses or language-conditioned models. These results demonstrate that properly integrated semantic supervision provides powerful inductive biases for more efficient robot learning.

Figures (5)

• Figure 1: Overview of our approach: Top: We enhance the normal Imitation Learning approach by allowing the user to specify which concepts are in an episode. Bottom: We integrate these concepts by extending the ACT architecture to include a Concept Transformer that aligns its attention mechanism with the given concepts. This alignment cost can then be included in the total loss. Red indicates changes.
• Figure 2: Experimental hardware configuration showing different camera viewpoints and the bilateral robotic system
• Figure 3: Sample efficiency results on task 1. Real robot evaluation performance is measured on different sizes of the dataset (with respect to the maximum size) on 10 test episodes (higher is better). Each method was trained with 10 random seeds; bold lines represent means, shaded areas show variance.
• Figure 4: Sample efficiency results on task 1. Real robot evaluation performance is measured on different sizes of the dataset (with respect to the maximum size) on 10 test episodes (higher is better). Each method was trained with 10 random seeds; bold lines represent the means, and shaded areas show the variance.
• Figure 5: Learning efficiency results comparing convergence speed across methods. (a) Test loss on 20% holdout set throughout training (lower is better). (b) Real robot evaluation performance is measured every 3,000 training steps on 10 test episodes (higher is better). Each method was trained with 5 random seeds; bold lines represent means, shaded areas show variance.