Topic Aware Neural Response Generation

TL;DR

This work tackles open-domain chatbot response generation by injecting topic information into the Seq2Seq framework. It introduces TA-Seq2Seq, which uses a topic encoder, a joint attention mechanism, and a biased generation probability to favor topic words, with topic words learned from external sources via Twitter LDA. Empirical results on large-scale Chinese data show TA-Seq2Seq produces more informative, diverse, and topic-relevant responses and outperforms strong baselines in both automatic metrics and human judgments. The approach demonstrates the value of incorporating prior topical knowledge to enrich conversational AI.

Abstract

We consider incorporating topic information into the sequence-to-sequence framework to generate informative and interesting responses for chatbots. To this end, we propose a topic aware sequence-to-sequence (TA-Seq2Seq) model. The model utilizes topics to simulate prior knowledge of human that guides them to form informative and interesting responses in conversation, and leverages the topic information in generation by a joint attention mechanism and a biased generation probability. The joint attention mechanism summarizes the hidden vectors of an input message as context vectors by message attention, synthesizes topic vectors by topic attention from the topic words of the message obtained from a pre-trained LDA model, and let these vectors jointly affect the generation of words in decoding. To increase the possibility of topic words appearing in responses, the model modifies the generation probability of topic words by adding an extra probability item to bias the overall distribution. Empirical study on both automatic evaluation metrics and human annotations shows that TA-Seq2Seq can generate more informative and interesting responses, and significantly outperform the-state-of-the-art response generation models.

Figures (3)

• Figure 1: Graphic model of Twitter LDA
• Figure 2: Structure of TA-Seq2Seq
• Figure 3: Case study