Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Computational Representations of Character: An Austen Character Similarity Benchmark

Funing Yang, Carolyn Jane Anderson

TL;DR

This work presents AustenAlike, a benchmark suite of character similarities in Jane Austen’s novels, and uses AustenAlike to evaluate character features extracted using two pipelines, BookNLP and FanfictionNLP.

Abstract

Several systems have been developed to extract information about characters to aid computational analysis of English literature. We propose character similarity grouping as a holistic evaluation task for these pipelines. We present AustenAlike, a benchmark suite of character similarities in Jane Austen's novels. Our benchmark draws on three notions of character similarity: a structurally defined notion of similarity; a socially defined notion of similarity; and an expert defined set extracted from literary criticism. We use AustenAlike to evaluate character features extracted using two pipelines, BookNLP and FanfictionNLP. We build character representations from four kinds of features and compare them to the three AustenAlike benchmarks and to GPT-4 similarity rankings. We find that though computational representations capture some broad similarities based on shared social and narrative roles, the expert pairings in our third benchmark are challenging for all systems, highlighting the subtler aspects of similarity noted by human readers.

Evaluating Computational Representations of Character: An Austen Character Similarity Benchmark

TL;DR

This work presents AustenAlike, a benchmark suite of character similarities in Jane Austen’s novels, and uses AustenAlike to evaluate character features extracted using two pipelines, BookNLP and FanfictionNLP.

Abstract

Several systems have been developed to extract information about characters to aid computational analysis of English literature. We propose character similarity grouping as a holistic evaluation task for these pipelines. We present AustenAlike, a benchmark suite of character similarities in Jane Austen's novels. Our benchmark draws on three notions of character similarity: a structurally defined notion of similarity; a socially defined notion of similarity; and an expert defined set extracted from literary criticism. We use AustenAlike to evaluate character features extracted using two pipelines, BookNLP and FanfictionNLP. We build character representations from four kinds of features and compare them to the three AustenAlike benchmarks and to GPT-4 similarity rankings. We find that though computational representations capture some broad similarities based on shared social and narrative roles, the expert pairings in our third benchmark are challenging for all systems, highlighting the subtler aspects of similarity noted by human readers.
Paper Structure (56 sections, 3 figures, 9 tables)

This paper contains 56 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Representing Literary Characters
  3. Character mentions
  4. Character features
  5. Character models
  6. A Three-Part Benchmark for Evaluating Character Similarity
  7. Social Characteristics
  8. Rank
  9. Wealth
  10. Gender
  11. Age
  12. Marital status
  13. Narrative Roles
  14. Wisdom-of-the-Experts Character Pairs
  15. Building Computational Representations of Character
  16. ...and 41 more sections

Figures (3)

  • Figure 1: Example character from AustenAlike
  • Figure 2: Narrative Role Benchmark: Mean cosine similarities between same-group characters and other characters by representation type.
  • Figure 3: Social Benchmark: average differences in cosine similarity between same-group characters and other characters by character representation and social role group.