Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models Meet Harry Potter: A Bilingual Dataset for Aligning Dialogue Agents with Characters

Nuo Chen, Yan Wang, Haiyun Jiang, Deng Cai, Yuhan Li, Ziyang Chen, Longyue Wang, Jia Li

TL;DR

The paper introduces HPD, a bilingual dataset that pairs Harry Potter dialogues with time-sensitive scene context, attributes, and relation annotations to study character-aligned dialogue agents. It formalizes the alignment task, details meticulous dataset construction (dialogues, scenes, and 13 attributes across 12 relations for 113 characters), and provides a robust test set designed to evaluate generation and retrieval approaches. Experimental results show that rich-background prompts improve alignment, but even top models fall short of human experts, underscoring substantial room for improvement. HPD offers a rich, dynamic benchmark for future work in character-aware dialogue systems and related tasks like sentiment analysis and reading comprehension in story worlds.

Abstract

In recent years, Dialogue-style Large Language Models (LLMs) such as ChatGPT and GPT4 have demonstrated immense potential in constructing open-domain dialogue agents. However, aligning these agents with specific characters or individuals remains a considerable challenge due to the complexities of character representation and the lack of comprehensive annotations. In this paper, we introduce the Harry Potter Dialogue (HPD) dataset, designed to advance the study of dialogue agents and character alignment. The dataset encompasses all dialogue sessions (in both English and Chinese) from the Harry Potter series and is annotated with vital background information, including dialogue scenes, speakers, character relationships, and attributes. These extensive annotations may empower LLMs to unlock character-driven dialogue capabilities. Furthermore, it can serve as a universal benchmark for evaluating how well can a LLM aligning with a specific character. We benchmark LLMs on HPD using both fine-tuning and in-context learning settings. Evaluation results reveal that although there is substantial room for improvement in generating high-quality, character-aligned responses, the proposed dataset is valuable in guiding models toward responses that better align with the character of Harry Potter.

Large Language Models Meet Harry Potter: A Bilingual Dataset for Aligning Dialogue Agents with Characters

TL;DR

The paper introduces HPD, a bilingual dataset that pairs Harry Potter dialogues with time-sensitive scene context, attributes, and relation annotations to study character-aligned dialogue agents. It formalizes the alignment task, details meticulous dataset construction (dialogues, scenes, and 13 attributes across 12 relations for 113 characters), and provides a robust test set designed to evaluate generation and retrieval approaches. Experimental results show that rich-background prompts improve alignment, but even top models fall short of human experts, underscoring substantial room for improvement. HPD offers a rich, dynamic benchmark for future work in character-aware dialogue systems and related tasks like sentiment analysis and reading comprehension in story worlds.

Abstract

In recent years, Dialogue-style Large Language Models (LLMs) such as ChatGPT and GPT4 have demonstrated immense potential in constructing open-domain dialogue agents. However, aligning these agents with specific characters or individuals remains a considerable challenge due to the complexities of character representation and the lack of comprehensive annotations. In this paper, we introduce the Harry Potter Dialogue (HPD) dataset, designed to advance the study of dialogue agents and character alignment. The dataset encompasses all dialogue sessions (in both English and Chinese) from the Harry Potter series and is annotated with vital background information, including dialogue scenes, speakers, character relationships, and attributes. These extensive annotations may empower LLMs to unlock character-driven dialogue capabilities. Furthermore, it can serve as a universal benchmark for evaluating how well can a LLM aligning with a specific character. We benchmark LLMs on HPD using both fine-tuning and in-context learning settings. Evaluation results reveal that although there is substantial room for improvement in generating high-quality, character-aligned responses, the proposed dataset is valuable in guiding models toward responses that better align with the character of Harry Potter.
Paper Structure (35 sections, 1 equation, 5 figures, 7 tables)

This paper contains 35 sections, 1 equation, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Task Definition
  3. Dataset Construction
  4. Dialogue Construction
  5. Training Set
  6. Test Set
  7. Scenes Construction
  8. Attributes and Relations Construction
  9. Affection Definition
  10. Familiarity Definition
  11. Annotation
  12. Claim
  13. Data Statistics
  14. Experiments
  15. Baselines
  16. ...and 20 more sections

Figures (5)

  • Figure 1: A conversation between Harry and Hermione selected from Book 4-Chapter 19 in the Harry Potter Series. In this example, we present the scene, timing of the conversation, relations and attributes of speakers. Texts in red refer to the expected response.
  • Figure 2: Data examples from two different timings: Book 1-Chapter 12 and Book 6-Chapter 7. Words in color denote the changed relations and attributes.
  • Figure 3: Affection Definition and Examples.
  • Figure 4: Familiarity Definition and Examples.
  • Figure 5: Average ranks of generated texts in terms of human evaluation and GPT-4 evaluations.