Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Says Who? Effective Zero-Shot Annotation of Focalization

Rebecca M. M. Hicke, Yuri Bizzoni, Pascale Feldkamp, Ross Deans Kristensen-McLachlan

TL;DR

This paper tackles the challenge of annotating focalization in narrative discourse using zero-shot large language models. It systematically compares multiple model families and baselines on 256 excerpts from 16 Stephen King novels, introducing a rigorous human annotation protocol and a training set, and demonstrates that GPT-4o achieves near-human performance (F1 ≈ 84.8%). The study also shows that GPT-derived confidence aligns with annotation difficulty and leverages log probabilities to gauge uncertainty. At scale, focalization annotations enable structural comparisons across novels and reveal links between focalization modes and sensorimotor linguistic features, illustrating the practical value of LLM-based annotation for computational literary studies.

Abstract

Focalization describes the way in which access to narrative information is restricted or controlled based on the knowledge available to knowledge of the narrator. It is encoded via a wide range of lexico-grammatical features and is subject to reader interpretation. Even trained annotators frequently disagree on correct labels, suggesting this task is both qualitatively and computationally challenging. In this work, we test how well five contemporary large language model (LLM) families and two baselines perform when annotating short literary excerpts for focalization. Despite the challenging nature of the task, we find that LLMs show comparable performance to trained human annotators, with GPT-4o achieving an average F1 of 84.79%. Further, we demonstrate that the log probabilities output by GPT-family models frequently reflect the difficulty of annotating particular excerpts. Finally, we provide a case study analyzing sixteen Stephen King novels, demonstrating the usefulness of this approach for computational literary studies and the insights gleaned from examining focalization at scale.

Says Who? Effective Zero-Shot Annotation of Focalization

TL;DR

This paper tackles the challenge of annotating focalization in narrative discourse using zero-shot large language models. It systematically compares multiple model families and baselines on 256 excerpts from 16 Stephen King novels, introducing a rigorous human annotation protocol and a training set, and demonstrates that GPT-4o achieves near-human performance (F1 ≈ 84.8%). The study also shows that GPT-derived confidence aligns with annotation difficulty and leverages log probabilities to gauge uncertainty. At scale, focalization annotations enable structural comparisons across novels and reveal links between focalization modes and sensorimotor linguistic features, illustrating the practical value of LLM-based annotation for computational literary studies.

Abstract

Focalization describes the way in which access to narrative information is restricted or controlled based on the knowledge available to knowledge of the narrator. It is encoded via a wide range of lexico-grammatical features and is subject to reader interpretation. Even trained annotators frequently disagree on correct labels, suggesting this task is both qualitatively and computationally challenging. In this work, we test how well five contemporary large language model (LLM) families and two baselines perform when annotating short literary excerpts for focalization. Despite the challenging nature of the task, we find that LLMs show comparable performance to trained human annotators, with GPT-4o achieving an average F1 of 84.79%. Further, we demonstrate that the log probabilities output by GPT-family models frequently reflect the difficulty of annotating particular excerpts. Finally, we provide a case study analyzing sixteen Stephen King novels, demonstrating the usefulness of this approach for computational literary studies and the insights gleaned from examining focalization at scale.
Paper Structure (16 sections, 1 figure, 6 tables)

This paper contains 16 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methods
  4. Data
  5. Human Annotation
  6. Automated Annotation
  7. Annotation Accuracy and Agreement
  8. Human Performance
  9. Model Performance
  10. Comparing Novel Structures
  11. Links to Linguistic Features
  12. Conclusion
  13. Corpus Contents
  14. Fine-Tuning Parameters
  15. Prompt Variants
  16. ...and 1 more sections

Figures (1)

  • Figure 1: The prompt used for all zero-shot experiments. The definitions for each mode of focalization are adapted from those given in chen2023. The paragraph to be analyzed is inserted after the EXCERPT: heading.