Retrieval-Augmented Hierarchical in-Context Reinforcement Learning and Hindsight Modular Reflections for Task Planning with LLMs

TL;DR

Retrieval-Augmented Hierarchical in-context reinforcement Learning (RAHL), a novel framework that decomposes complex tasks into sub-tasks using an LLM-based high-level policy, in which a complex task is decomposed into sub-tasks by a high-level policy on-the-fly.

Abstract

Large Language Models (LLMs) have demonstrated remarkable abilities in various language tasks, making them promising candidates for decision-making in robotics. Inspired by Hierarchical Reinforcement Learning (HRL), we propose Retrieval-Augmented in-context reinforcement Learning (RAHL), a novel framework that decomposes complex tasks into sub-tasks using an LLM-based high-level policy, in which a complex task is decomposed into sub-tasks by a high-level policy on-the-fly. The sub-tasks, defined by goals, are assigned to the low-level policy to complete. To improve the agent's performance in multi-episode execution, we propose Hindsight Modular Reflection (HMR), where, instead of reflecting on the full trajectory, we let the agent reflect on shorter sub-trajectories to improve reflection efficiency. We evaluated the decision-making ability of the proposed RAHL in three benchmark environments--ALFWorld, Webshop, and HotpotQA. The results show that RAHL can achieve an improvement in performance in 9%, 42%, and 10% in 5 episodes of execution in strong baselines. Furthermore, we also implemented RAHL on the Boston Dynamics SPOT robot. The experiment shows that the robot can scan the environment, find entrances, and navigate to new rooms controlled by the LLM policy.

Figures (5)

• Figure 1: (a) Demonstration of a typical task in ALFWorld. The high-level reflection attempts to correct the errors made by the high-level policy, while the low-level reflection corrects the errors from the low-level policy. (b) Flow diagram of the proposed Retrieval-Augmented Hierarchical in-context reinforcement Learning (RAHL) and Hindsight Modular Reflection (HMR).
• Figure 2: The decision-making process of RAHL.
• Figure 3: Success rate over five episodes in three datasets/environments using GPT-3.5-turbo. The results and confidence intervals are obtained over ten runs.
• Figure 4: Success rates in percentage obtained with different LLMs. GPT-3.5 and GPT-4 indicate the decision-making and reflection are performed with the same type of LLMs, while GPT-3.5 with GPT-4 HMR indicates GPT-3.5 is used for decision-making while GPT-4 is used for reflection. The results and confidence intervals are obtained over ten runs.
• Figure 5: (a) The diagram of the system designed for the experiment with Boston Dynamics SPOT. (b) Robot (green spearhead with the head pointing in the camera's direction) trajectory in the rooms to locate the target person (red dot).