PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model

TL;DR

PanGu-Bot tackles data-efficient open-domain Chinese dialogue by inheriting a large pre-trained language model (PanGu-α) rather than training from scratch. By leveraging PanGu-α and a comparatively small, high-quality dialogue corpus, PanGu-Bot demonstrates improvements in response quality, knowledge correctness, and safety relative to state-of-the-art Chinese dialogue systems. The work highlights the practical potential of transferring broad language capabilities from PLMs to dialogue tasks with reduced data and compute, while acknowledging ongoing challenges in safety and reliability. The authors provide detailed model configurations and training strategies, with code release planned to support reproducibility.

Abstract

In this paper, we introduce PanGu-Bot, a Chinese pre-trained open-domain dialogue generation model based on a large pre-trained language model (PLM) PANGU-alpha (Zeng et al.,2021). Different from other pre-trained dialogue models trained over a massive amount of dialogue data from scratch, we aim to build a powerful dialogue model with relatively fewer data and computation costs by inheriting valuable language capabilities and knowledge from PLMs. To this end, we train PanGu-Bot from the large PLM PANGU-alpha, which has been proven well-performed on a variety of Chinese natural language tasks. We investigate different aspects of responses generated by PanGu-Bot, including response quality, knowledge, and safety. We show that PanGu-Bot outperforms state-of-the-art Chinese dialogue systems (CDIALGPT (Wang et al., 2020), EVA (Zhou et al., 2021), EVA2.0 (Gu et al., 2022)) w.r.t. the above three aspects. We also demonstrate that PanGu-Bot can be easily deployed to generate emotional responses without further training. Throughout our empirical analysis, we also point out that the PanGu-Bot response quality, knowledge correctness, and safety are still far from perfect, and further explorations are indispensable to building reliable and smart dialogue systems. Our model and code will be available at https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/PanGu-Bot soon.

Figures (1)

• Figure 1: Illustration of model training with multiple sessions. represents attention blocked by the attention mask. Position embedding ids are reset between sessions, and attention masks are also reset between sessions to prevent interference from utterances in previous sessions.