LLM-based Interactive Imitation Learning for Robotic Manipulation

TL;DR

The paper introduces LLM-iTeach, an interactive imitation learning framework that replaces human teachers with a Large Language Model to guide robotic manipulation tasks. By encoding the LLM’s reasoning into a precomputed CodePolicy via hierarchical prompting and delivering evaluative/corrective feedback through a similarity-based mechanism, the approach trains an agent with a Gaussian policy while weighting feedback to emphasize informative corrections. Empirical results on RLBench tasks show LLM-iTeach achieving performance comparable to or better than state-of-the-art human-guided CEILing and surpassing BC, with demonstrated generalization to additional tasks. The work highlights the practical potential of LLMs as cost-effective, scalable teachers in interactive learning for robotics, while noting limitations such as the LLM’s partial observability and the need for ground-truth data for feedback generation.

Abstract

Recent advancements in machine learning provide methods to train autonomous agents capable of handling the increasing complexity of sequential decision-making in robotics. Imitation Learning (IL) is a prominent approach, where agents learn to control robots based on human demonstrations. However, IL commonly suffers from violating the independent and identically distributed (i.i.d) assumption in robotic tasks. Interactive Imitation Learning (IIL) achieves improved performance by allowing agents to learn from interactive feedback from human teachers. Despite these improvements, both approaches come with significant costs due to the necessity of human involvement. Leveraging the emergent capabilities of Large Language Models (LLMs) in reasoning and generating human-like responses, we introduce LLM-iTeach -- a novel IIL framework that utilizes an LLM as an interactive teacher to enhance agent performance while alleviating the dependence on human resources. Firstly, LLM-iTeach uses a hierarchical prompting strategy that guides the LLM in generating a policy in Python code. Then, with a designed similarity-based feedback mechanism, LLM-iTeach provides corrective and evaluative feedback interactively during the agent's training. We evaluate LLM-iTeach against baseline methods such as Behavior Cloning (BC), an IL method, and CEILing, a state-of-the-art IIL method using a human teacher, on various robotic manipulation tasks. Our results demonstrate that LLM-iTeach surpasses BC in the success rate and achieves or even outscores that of CEILing, highlighting the potential of LLMs as cost-effective, human-like teachers in interactive learning environments. We further demonstrate the method's potential for generalization by evaluating it on additional tasks. The code and prompts are provided at: https://github.com/Tubicor/LLM-iTeach.

Figures (9)

• Figure 1: Comparison of the General IIL framework and our proposed LLM-iTeach. In IIL, a human teacher observes the agent's actions and provides timely feedback. In contrast, LLM-iTeach first prompts the LLM to encapsulate its reasoning hierarchically into CodePolicy, and then provides feedback to the agent in an evaluative or corrective manner through the similarity-checking mechanism designed in LLM-iTeach.
• Figure 2: Abstraction of consecutive construction of CodePolicy with hierarchical prompting. The green colored lines are the instructions of the prompts, and the orange lines are the corresponding generation of the LLM. The two hierarchical levels of the prompting build on top of each other and are composed of the CodePolicy.
• Figure 3: The model architecture for the learning agent in LLM-iTeach.
• Figure 4: The tasks on the top row are shared with CEILing. The tasks on the bottom row are additional tasks. The screenshots are taken from the Graphical User Interface of RLBench RLBench.
• Figure 5: The average success rate of BC and LLM-iTeach for the number of episodes used in the training phase. Additionally, the performance of the LLM Teacher is given.