Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access

TL;DR

The paper addresses end-to-end training of dialogue agents that access knowledge bases by replacing non-differentiable symbolic queries with a differentiable Soft-KB lookup that maintains a posterior over entities from slot-level beliefs.KB-InfoBot combines belief tracking (hand-crafted and neural), a differentiable soft retrieval mechanism, and both hand-crafted and neural dialogue policies, enabling reinforcement learning and imitation learning to optimize task success.Experiments on a movie-domain EC-KB show Soft-KB-based methods outperform hard-symbolic baselines in simulated and some real-user settings, while an end-to-end variant demonstrates strong learning capacity but tends to overfit and require data for robust generalization.The work argues for a pragmatic deployment strategy that starts with RL-based Soft-KB, then transitions toward personalized end-to-end models as experience accumulates, highlighting practical implications for building information-access agents.

Abstract

This paper proposes KB-InfoBot -- a multi-turn dialogue agent which helps users search Knowledge Bases (KBs) without composing complicated queries. Such goal-oriented dialogue agents typically need to interact with an external database to access real-world knowledge. Previous systems achieved this by issuing a symbolic query to the KB to retrieve entries based on their attributes. However, such symbolic operations break the differentiability of the system and prevent end-to-end training of neural dialogue agents. In this paper, we address this limitation by replacing symbolic queries with an induced "soft" posterior distribution over the KB that indicates which entities the user is interested in. Integrating the soft retrieval process with a reinforcement learner leads to higher task success rate and reward in both simulations and against real users. We also present a fully neural end-to-end agent, trained entirely from user feedback, and discuss its application towards personalized dialogue agents. The source code is available at https://github.com/MiuLab/KB-InfoBot.

Figures (5)

• Figure 1: An interaction between a user looking for a movie and the KB-InfoBot. An entity-centric knowledge base is shown above the KB-InfoBot (missing values denoted by X).
• Figure 2: High-level overview of the end-to-end KB-InfoBot. Components with trainable parameters are highlighted in gray.
• Figure 3: Performance of KB-InfoBot versions when tested against real users. Left: Success rate, with the number of test dialogues indicated on each bar, and the p-values from a two-sided permutation test. Right: Distribution of the number of turns in each dialogue (differences in mean are significant with $p<0.01$).
• Figure 4: Sample dialogues between users and the KB-InfoBot (RL-Soft version). Each turn begins with a user utterance followed by the agent response. Rank denotes the rank of the target movie in the KB-posterior after each turn.
• Figure 5: Average rewards against simulator as temperature of softmax in NLG output is increased. Higher temperature leads to more noise in output. Average over $5000$ simulations after selecting the best model during training.