Training Dialogue Systems by AI Feedback for Improving Overall Dialogue Impression

TL;DR

This work tackles the challenge of improving not just individual dialogue turns but the overall dialogue impression (consistency, personality, empathy) in LLM-based systems. It introduces a reward-estimator trained via supervised fine-tuning to assess 12 impression metrics from dialogue context and responses, enabling multi-metric evaluation beyond zero-shot prompts. The authors compare prompting-based rewards with SFT-trained reward signals and demonstrate that a 7B SFT reward-estimator achieves strong alignment with human judgments, which is then used to train dialogue models via PPO and DPO, with DPO delivering the best automatic and human performance. The results show that AI feedback-guided fine-tuning improves both the naturalness and alignment with dialogue-impression values, supporting scalable enhancement of conversational agents while highlighting issues like reward bias toward natural but dull outputs and the need for diversity-aware strategies.

Abstract

To improve user engagement during conversations with dialogue systems, we must improve individual dialogue responses and dialogue impressions such as consistency, personality, and empathy throughout the entire dialogue. While such dialogue systems have been developing rapidly with the help of large language models (LLMs), reinforcement learning from AI feedback (RLAIF) has attracted attention to align LLM-based dialogue models for such dialogue impressions. In RLAIF, a reward model based on another LLM is used to create a training signal for an LLM-based dialogue model using zero-shot/few-shot prompting techniques. However, evaluating an entire dialogue only by prompting LLMs is challenging. In this study, the supervised fine-tuning (SFT) of LLMs prepared reward models corresponding to 12 metrics related to the impression of the entire dialogue for evaluating dialogue responses. We tuned our dialogue models using the reward model signals as feedback to improve the impression of the system. The results of automatic and human evaluations showed that tuning the dialogue model using our reward model corresponding to dialogue impression improved the evaluation of individual metrics and the naturalness of the dialogue response.

Figures (1)

• Figure 1: Training of reward model and implementation of PPO and DPO, where $C_i$ is dialogue context, $R_i$ is generated response, the evaluation score $S_{i, E_j}$ corresponding to a metric $E_j$ as inputs