Mutual Enhancement of Large Language and Reinforcement Learning Models through Bi-Directional Feedback Mechanisms: A Planning Case Study

TL;DR

The paper addresses how to effectively harness LLMs to assist RL planning while enabling RL feedback to refine LLM guidance through a bi-directional teacher-student framework in a cooperative setting. It formalizes an iterative, feedback-driven collaboration where LLMs provide high-level instructions to accelerate RL exploration and RL agents critique and improve subsequent LLM guidance. Empirical evaluation on the BabyAI GoToRedBallNoDists-v0 task within the Lamorel framework shows higher performance and faster convergence than the baseline using an 80M-parameter LLM, across 40 and 2100 iteration budgets. The work demonstrates a practical, robust planning paradigm with potential for safe deployment in imperfect-information environments and lays groundwork for extending to more complex, real-world tasks.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities for reinforcement learning (RL) models, such as planning and reasoning capabilities. However, the problems of LLMs and RL model collaboration still need to be solved. In this study, we employ a teacher-student learning framework to tackle these problems, specifically by offering feedback for LLMs using RL models and providing high-level information for RL models with LLMs in a cooperative multi-agent setting. Within this framework, the LLM acts as a teacher, while the RL model acts as a student. The two agents cooperatively assist each other through a process of recursive help, such as "I help you help I help." The LLM agent supplies abstract information to the RL agent, enabling efficient exploration and policy improvement. In turn, the RL agent offers feedback to the LLM agent, providing valuable, real-time information that helps generate more useful tokens. This bi-directional feedback loop promotes optimization, exploration, and mutual improvement for both agents, enabling them to accomplish increasingly challenging tasks. Remarkably, we propose a practical algorithm to address the problem and conduct empirical experiments to evaluate the effectiveness of our method.

Figures (2)

• Figure 1: An LLM and an RL model collaboratively engage with an environment to accomplish complex tasks, facilitating bi-directional feedback throughout the process.
• Figure 2: Experiments on BabyAI tasks babyaiiclr19 with 40 (a) and 2100 (b) iteration steps.