Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations

TL;DR

The paper tackles goal-directed dialogue by proposing an imagination engine (IE) that uses LLMs to generate diverse, task-relevant but suboptimal human–agent conversations from a task description. These imagined dialogues are then used to train an offline RL agent (via ILQL) that optimizes multi-turn conversational goals, addressing information gathering and personalization. Through both a human user study and large-scale simulations across instruction and preference elicitation tasks, the IE+RL approach outperforms prompting-based baselines and imitation-learning on imagined data, demonstrating robust, zero-shot capability. This work demonstrates a practical pathway to scalable, personalized, goal-directed dialogue without expensive online or human data collection, by leveraging LLM-generated data to train specialized RL agents.

Abstract

Large language models (LLMs) have emerged as powerful and general solutions to many natural language tasks. However, many of the most important applications of language generation are interactive, where an agent has to talk to a person to reach a desired outcome. For example, a teacher might try to understand their student's current comprehension level to tailor their instruction accordingly, and a travel agent might ask questions of their customer to understand their preferences in order to recommend activities they might enjoy. LLMs trained with supervised fine-tuning or "single-step" RL, as with standard RLHF, might struggle which tasks that require such goal-directed behavior, since they are not trained to optimize for overall conversational outcomes after multiple turns of interaction. In this work, we explore a new method for adapting LLMs with RL for such goal-directed dialogue. Our key insight is that, though LLMs might not effectively solve goal-directed dialogue tasks out of the box, they can provide useful data for solving such tasks by simulating suboptimal but human-like behaviors. Given a textual description of a goal-directed dialogue task, we leverage LLMs to sample diverse synthetic rollouts of hypothetical in-domain human-human interactions. Our algorithm then utilizes this dataset with offline reinforcement learning to train an interactive conversational agent that can optimize goal-directed objectives over multiple turns. In effect, the LLM produces examples of possible interactions, and RL then processes these examples to learn to perform more optimal interactions. Empirically, we show that our proposed approach achieves state-of-the-art performance in various goal-directed dialogue tasks that include teaching and preference elicitation.

Figures (6)

• Figure 1: Illustrative example of how existing LLMs behave when prompted to act as travel agents vs. how effective travel agents should behave.
• Figure 2: Diagram illustrating our proposed approach, where an imagined dataset of dialogues between humans and a potentially suboptimal agent is synthesized by our imagination engine, then used to train a downstream RL agent. Blue boxes indicate handcrafted quantities.
• Figure 3: Comparison of dialogues between GPT and IE+RL agents in instruction task. The IE+RL agent exhibits a much more intelligent strategy of asking incremental questions.
• Figure 4: Comparison of dialogues between GPT and IE+RL agents in preference elicitation task. The IE+RL agent adapts to the user giving vague responses by asking questions with more narrow options.
• Figure 5: Comparison of dialogues between IE+FBC and IE+RL agents in instruction task. The IE+RL agent is much more effective at responding to the user being confused.