Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learning
Ziming Li, Julia Kiseleva, Maarten de Rijke
TL;DR
The paper tackles open-domain dialogue generation by addressing unstable and sparse discriminator rewards that hinder generator quality. It introduces two adversarial frameworks: DG-AIL, which adds causal-entropy regularization to adversarial imitation learning, and DG-AIRL, which learns a structured word-level reward function using adversarial inverse reinforcement learning. The DG-AIRL approach yields a more accurate reward signal and, combined with entropy regularization, leads to higher-quality and more contextually appropriate responses, as shown by embedding metrics and human judgments outperforming state-of-the-art baselines. The work advances reward modeling in dialogue systems and demonstrates practical gains in generating engaging and coherent open-domain conversations.
Abstract
The performance of adversarial dialogue generation models relies on the quality of the reward signal produced by the discriminator. The reward signal from a poor discriminator can be very sparse and unstable, which may lead the generator to fall into a local optimum or to produce nonsense replies. To alleviate the first problem, we first extend a recently proposed adversarial dialogue generation method to an adversarial imitation learning solution. Then, in the framework of adversarial inverse reinforcement learning, we propose a new reward model for dialogue generation that can provide a more accurate and precise reward signal for generator training. We evaluate the performance of the resulting model with automatic metrics and human evaluations in two annotation settings. Our experimental results demonstrate that our model can generate more high-quality responses and achieve higher overall performance than the state-of-the-art.