Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works

Xinfeng Yuan, Siyu Yuan, Yuhan Cui, Tianhe Lin, Xintao Wang, Rui Xu, Jiangjie Chen, Deqing Yang

TL;DR

The paper addresses how to rigorously evaluate LLMs understanding of fictional characters by proposing a structured character profiling task. It introduces CroSS, a high quality dataset derived from SuperSummary, and defines four profiling dimensions—attributes, relationships, events, and personality—alongside three long-context summarization methods. Through intrinsic Factual Consistency Examination and extrinsic Motivation Recognition tasks, the work demonstrates that LLMs, especially GPT-4 Turbo, can produce coherent character profiles while highlighting where information loss occurs, particularly in events, and the value of complete-book one-shot summarization. The results establish a meaningful link between factual profile accuracy and downstream character understanding, while acknowledging limitations such as potential data leakage and evaluator biases, and charting a path for future improvements in multi-dimensional character profiling for RPAs and related applications.

Abstract

Large language models (LLMs) have demonstrated impressive performance and spurred numerous AI applications, in which role-playing agents (RPAs) are particularly popular, especially for fictional characters. The prerequisite for these RPAs lies in the capability of LLMs to understand characters from fictional works. Previous efforts have evaluated this capability via basic classification tasks or characteristic imitation, failing to capture the nuanced character understanding with LLMs. In this paper, we propose evaluating LLMs' character understanding capability via the character profiling task, i.e., summarizing character profiles from corresponding materials, a widely adopted yet understudied practice for RPA development. Specifically, we construct the CroSS dataset from literature experts and assess the generated profiles by comparing them with ground truth references and evaluating their applicability in downstream tasks. Our experiments, which cover various summarization methods and LLMs, have yielded promising results. These results strongly validate the character understanding capability of LLMs. Resources are available at https://github.com/Joanna0123/character_profiling.

Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works

TL;DR

The paper addresses how to rigorously evaluate LLMs understanding of fictional characters by proposing a structured character profiling task. It introduces CroSS, a high quality dataset derived from SuperSummary, and defines four profiling dimensions—attributes, relationships, events, and personality—alongside three long-context summarization methods. Through intrinsic Factual Consistency Examination and extrinsic Motivation Recognition tasks, the work demonstrates that LLMs, especially GPT-4 Turbo, can produce coherent character profiles while highlighting where information loss occurs, particularly in events, and the value of complete-book one-shot summarization. The results establish a meaningful link between factual profile accuracy and downstream character understanding, while acknowledging limitations such as potential data leakage and evaluator biases, and charting a path for future improvements in multi-dimensional character profiling for RPAs and related applications.

Abstract

Large language models (LLMs) have demonstrated impressive performance and spurred numerous AI applications, in which role-playing agents (RPAs) are particularly popular, especially for fictional characters. The prerequisite for these RPAs lies in the capability of LLMs to understand characters from fictional works. Previous efforts have evaluated this capability via basic classification tasks or characteristic imitation, failing to capture the nuanced character understanding with LLMs. In this paper, we propose evaluating LLMs' character understanding capability via the character profiling task, i.e., summarizing character profiles from corresponding materials, a widely adopted yet understudied practice for RPA development. Specifically, we construct the CroSS dataset from literature experts and assess the generated profiles by comparing them with ground truth references and evaluating their applicability in downstream tasks. Our experiments, which cover various summarization methods and LLMs, have yielded promising results. These results strongly validate the character understanding capability of LLMs. Resources are available at https://github.com/Joanna0123/character_profiling.
Paper Structure (67 sections, 3 figures, 16 tables)

This paper contains 67 sections, 3 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Character Role-Playing
  4. Motivation Analysis & Character Understanding
  5. Character Profiling Framework
  6. Task Formulation
  7. Character Profile Dimensions
  8. Attributes
  9. Relationships
  10. Events
  11. Personality
  12. Summarization Methods
  13. Hierarchical Merging
  14. Incremental Updating
  15. Summarizing in One Go
  16. ...and 52 more sections

Figures (3)

  • Figure 1: An overview of character profiling with LLMs and the two evaluation tasks we proposed, including factual consistency examination and motivation recognition.
  • Figure 2: The three methods of long context processing for LLM-based character profiling.
  • Figure 3: Average Consistency Score of books in "Best Books in {#the year}" list in goodreads in different years.