ELEMENTAL: Interactive Learning from Demonstrations and Vision-Language Models for Reward Design in Robotics

TL;DR

ELEMENTAL addresses the challenge of reward design in robotic reinforcement learning by integrating vision-language models with learning from demonstration and inverse reinforcement learning. It uses a three-phase workflow: VLM-driven initial feature extraction from textual task descriptions and visual demonstrations, IRL-based reward and policy optimization, and an automatic self-reflection loop that refines features based on trajectory-feature discrepancies. This combination reduces reward specification ambiguity and improves generalization to out-of-distribution tasks, achieving substantial gains on IsaacGym benchmarks. While trading off runtime due to environment rollouts, ELEMENTAL demonstrates robust performance and adaptable reward design that better aligns robot behavior with human intent in diverse robotic tasks.

Abstract

Reinforcement learning (RL) has demonstrated compelling performance in robotic tasks, but its success often hinges on the design of complex, ad hoc reward functions. Researchers have explored how Large Language Models (LLMs) could enable non-expert users to specify reward functions more easily. However, LLMs struggle to balance the importance of different features, generalize poorly to out-of-distribution robotic tasks, and cannot represent the problem properly with only text-based descriptions. To address these challenges, we propose ELEMENTAL (intEractive LEarning froM dEmoNstraTion And Language), a novel framework that combines natural language guidance with visual user demonstrations to align robot behavior with user intentions better. By incorporating visual inputs, ELEMENTAL overcomes the limitations of text-only task specifications, while leveraging inverse reinforcement learning (IRL) to balance feature weights and match the demonstrated behaviors optimally. ELEMENTAL also introduces an iterative feedback-loop through self-reflection to improve feature, reward, and policy learning. Our experiment results demonstrate that ELEMENTAL outperforms prior work by 42.3% on task success, and achieves 41.3% better generalization in out-of-distribution tasks, highlighting its robustness in LfD.

Figures (4)

• Figure 1: This figure illustrates the overall pipeline of ELEMENTAL. The process begins with an initial prompt to the VLM, which generates a draft of the feature function based on both textual descriptions and visual demonstrations. In the learning phase, ELEMENTAL infers the reward and policy from the drafted feature function and the demonstration. In the final phase, ELEMENTAL performs self-reflection by comparing the feature counts from the generated trajectory and the demonstration, again utilizing the drafted feature function. This self-reflection loop updates the feature function by feeding the results back to the VLM for iterative refinement.
• Figure 2: This figure illustrates the visual demonstrations for both locomotion and manipulation tasks. (a) shows an example from the Ant locomotion task, where superimposed images are used. For manipulation tasks, superimposed images can result in cluttered robot poses, so we use key frames as visual demonstration inputs instead. (b-e) present four key frames from the FrankaCabinet manipulation task.
• Figure 3: Approximate MaxEnt-IRL
• Figure 4: A comparison of the code executation rate between ELEMENTAL and EUREKA in three iterations of the algorithms.