GLIDE-RL: Grounded Language Instruction through DEmonstration in RL

TL;DR

GLIDE-RL tackles the challenge of grounding natural language in reinforcement learning under sparse rewards by introducing a three-agent Teacher-Instructor-Student framework that leverages curriculum learning with multiple teachers and an LLM-based instructor to generate synonymous natural language instructions for the student. The student operates in a goal-conditioned RL setting using a $Q(s,a)$ policy learned by a $D3QN$ with frame stacking, informed by language goals embedded with each instruction and augmented by a Behavioural Cloning loss, while teachers ensure the goals are reachable and instructors translate trajectories into language. Empirical evaluation in the BabyAI BossLevel demonstrates that a larger number of teachers and the BC component improve learning and generalization to unseen instructions, with the system capable of handling synonym instructions and maintaining non-zero-sum but adversarial-like rewards. Overall, GLIDE-RL shows that combining curriculum-driven demonstrations, natural language grounding, and demonstration-based training can significantly improve data efficiency and generalization for language-conditioned RL in sparse-reward environments.

Abstract

One of the final frontiers in the development of complex human - AI collaborative systems is the ability of AI agents to comprehend the natural language and perform tasks accordingly. However, training efficient Reinforcement Learning (RL) agents grounded in natural language has been a long-standing challenge due to the complexity and ambiguity of the language and sparsity of the rewards, among other factors. Several advances in reinforcement learning, curriculum learning, continual learning, language models have independently contributed to effective training of grounded agents in various environments. Leveraging these developments, we present a novel algorithm, Grounded Language Instruction through DEmonstration in RL (GLIDE-RL) that introduces a teacher-instructor-student curriculum learning framework for training an RL agent capable of following natural language instructions that can generalize to previously unseen language instructions. In this multi-agent framework, the teacher and the student agents learn simultaneously based on the student's current skill level. We further demonstrate the necessity for training the student agent with not just one, but multiple teacher agents. Experiments on a complex sparse reward environment validates the effectiveness of our proposed approach.

Figures (6)

• Figure 1: GLIDE-RL Algorithm has three independently functioning parts: The teacher, The instructor and The student. (1) The teacher is an agent that acts in the environment to do complex things and gets rewarded based on the performance of the student. (2) The instructor observes these actions, describes them in the form of events and converts them to the instruction format. (3) The student is the goal-conditioned agent that strives to reach the goals set by the teacher, as described and instructed by the instructor. (4) The instructor also checks if the student has triggered the same/similar events in the environment to mark a certain state as success/failure for the student.
• Figure 2: BabyAI BossLevel environment
• Figure 3: Success rate of different variations on the test set. The plot shows the mean and standard deviation over 5 seeds
• Figure 4: Q values of the synonymous events vs distance from the original event. Each blue dot ($\bullet$) represents one synonymous event for the original event. The red line represents the trend among the points
• Figure 5: Success rate on the test set with varying number of teachers. The plot shows mean and standard deviation over 5 seeds