DYSTIL: Dynamic Strategy Induction with Large Language Models for Reinforcement Learning

TL;DR

DYSTIL tackles the limitations of traditional RL from expert demonstrations by introducing a strategy-based framework that uses LLMs to induce textual, high-level strategies from demonstrations and advantage signals. It integrates these strategies into a novel four-part agent architecture and couples them with an on-policy PPO loop that dynamically updates strategies via advantage-informed prompts, then tests two copies of the agent to decide whether to adopt revised strategies. Across Minigrid Dynamic Obstacles and four BabyAI tasks, DYSTIL achieves higher mean returns and success rates than strong baselines and demonstrates superior sample efficiency, while also providing an interpretable view of policy evolution through textual strategies. This work advances language-grounded RL by combining strategy induction, dynamic distillation, and transparent policy reasoning to improve learning efficiency and generalization in complex, hierarchical tasks.

Abstract

Reinforcement learning from expert demonstrations has long remained a challenging research problem, and existing state-of-the-art methods using behavioral cloning plus further RL training often suffer from poor generalization, low sample efficiency, and poor model interpretability. Inspired by the strong reasoning abilities of large language models (LLMs), we propose a novel strategy-based reinforcement learning framework integrated with LLMs called DYnamic STrategy Induction with Llms for reinforcement learning (DYSTIL) to overcome these limitations. DYSTIL dynamically queries a strategy-generating LLM to induce textual strategies based on advantage estimations and expert demonstrations, and gradually internalizes induced strategies into the RL agent through policy optimization to improve its performance through boosting policy generalization and enhancing sample efficiency. It also provides a direct textual channel to observe and interpret the evolution of the policy's underlying strategies during training. We test DYSTIL over challenging RL environments from Minigrid and BabyAI, and empirically demonstrate that DYSTIL significantly outperforms state-of-the-art baseline methods by 17.75% in average success rate while also enjoying higher sample efficiency during the learning process.

Figures (8)

• Figure 1: An example strategy induction process from expert demonstrations in GPT-4o openai2024gpt4 for the Dynamic Obstacles RL environment from Minigrid MinigridMiniworld23. See Appendix \ref{['app:strategy_evolvement']} for the complete list of strategies induced in this example.
• Figure 2: The strategy-based model architecture of our DYSTIL RL agents.
• Figure 3: An overview of our proposed new modeling pipeline - Dynamic Strategy Induction with LLMs for Reinforcement Learning (DYSTIL). The steps depicted in green arrows corresponds to Initialization of the RL Agent Model, Initial Strategy Induction from Expert Demonstrations and Behavioral Cloning with Induced Strategies; the steps depicted in blue arrows corresponds to Experience Collection and Advantage Estimation and Induction of New Candidate List of Strategies; and the steps depicted in magenta arrows corresponds to Strategy-Integrated Proximal Policy Optimization.
• Figure 4: Comparison of sample efficiency between DYSTIL and GLAM on the four RL environments. The y-axis plots the maximum score of mean return evaluated on the validation set of environment configurations that the agent has achieved so far during the learning process, and x-axis plots the number of frames of training data that has been fed into the learning pipeline so far.
• Figure 5: The results of our ablation study.