Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Policy Networks with Two-Stage Training for Dialogue Systems

Mehdi Fatemi, Layla El Asri, Hannes Schulz, Jing He, Kaheer Suleman

TL;DR

The paper tackles data-efficient training of POMDP-based dialogue managers by evaluating deep RL models (DQN, DDQN, DA2C, and TDA2C) on DSTC2. It demonstrates that deep RL can learn effectively from original belief-action spaces, not just engineered summary spaces, and that a two-stage training approach—supervised pretraining followed by batch RL—significantly accelerates convergence and yields safer, valid actions. Key findings show faster convergence for deep RL versus GPSARSA on summary spaces, and strong performance with original spaces; two-stage training further enhances practical deployability. Overall, the approach enables end-to-end, data-efficient dialogue systems suitable for real-world deployment in restaurant-domain tasks.

Abstract

In this paper, we propose to use deep policy networks which are trained with an advantage actor-critic method for statistically optimised dialogue systems. First, we show that, on summary state and action spaces, deep Reinforcement Learning (RL) outperforms Gaussian Processes methods. Summary state and action spaces lead to good performance but require pre-engineering effort, RL knowledge, and domain expertise. In order to remove the need to define such summary spaces, we show that deep RL can also be trained efficiently on the original state and action spaces. Dialogue systems based on partially observable Markov decision processes are known to require many dialogues to train, which makes them unappealing for practical deployment. We show that a deep RL method based on an actor-critic architecture can exploit a small amount of data very efficiently. Indeed, with only a few hundred dialogues collected with a handcrafted policy, the actor-critic deep learner is considerably bootstrapped from a combination of supervised and batch RL. In addition, convergence to an optimal policy is significantly sped up compared to other deep RL methods initialized on the data with batch RL. All experiments are performed on a restaurant domain derived from the Dialogue State Tracking Challenge 2 (DSTC2) dataset.

Policy Networks with Two-Stage Training for Dialogue Systems

TL;DR

The paper tackles data-efficient training of POMDP-based dialogue managers by evaluating deep RL models (DQN, DDQN, DA2C, and TDA2C) on DSTC2. It demonstrates that deep RL can learn effectively from original belief-action spaces, not just engineered summary spaces, and that a two-stage training approach—supervised pretraining followed by batch RL—significantly accelerates convergence and yields safer, valid actions. Key findings show faster convergence for deep RL versus GPSARSA on summary spaces, and strong performance with original spaces; two-stage training further enhances practical deployability. Overall, the approach enables end-to-end, data-efficient dialogue systems suitable for real-world deployment in restaurant-domain tasks.

Abstract

In this paper, we propose to use deep policy networks which are trained with an advantage actor-critic method for statistically optimised dialogue systems. First, we show that, on summary state and action spaces, deep Reinforcement Learning (RL) outperforms Gaussian Processes methods. Summary state and action spaces lead to good performance but require pre-engineering effort, RL knowledge, and domain expertise. In order to remove the need to define such summary spaces, we show that deep RL can also be trained efficiently on the original state and action spaces. Dialogue systems based on partially observable Markov decision processes are known to require many dialogues to train, which makes them unappealing for practical deployment. We show that a deep RL method based on an actor-critic architecture can exploit a small amount of data very efficiently. Indeed, with only a few hundred dialogues collected with a handcrafted policy, the actor-critic deep learner is considerably bootstrapped from a combination of supervised and batch RL. In addition, convergence to an optimal policy is significantly sped up compared to other deep RL methods initialized on the data with batch RL. All experiments are performed on a restaurant domain derived from the Dialogue State Tracking Challenge 2 (DSTC2) dataset.

Paper Structure

This paper contains 16 sections, 1 theorem, 10 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Value-Based Deep Reinforcement Learning
  4. Deep Q Networks
  5. Double DQN: Overcoming Overestimation and Instability of DQN
  6. Policy Networks and Deep Advantage Actor-Critic (DA2C)
  7. Two-stage Training of the Policy Network
  8. Experiments
  9. Comparison of DQN and GPSARSA
  10. Experimental Protocol
  11. Results
  12. Comparison of the Deep RL Methods
  13. Experimental Protocol
  14. Results
  15. Concluding Remarks
  16. ...and 1 more sections

Key Result

Theorem 1

For any differentiable policy $\pi_{\theta}(b, a)$ and for the average reward or the start-state objective function, the policy gradient can be computed as

Figures (2)

  • Figure 1: Comparison of different algorithms on simulated dialogues, without any pre-training.
  • Figure 2: Comparison of different algorithms on simulated dialogues, with pre-training.

Theorems & Definitions (1)

  • Theorem 1: Policy Gradient