Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

S-VoCAL: A Dataset and Evaluation Framework for Inferring Speaking Voice Character Attributes in Literature

Abigail Berthe-Pardo, Gaspard Michel, Elena V. Epure, Christophe Cerisara

TL;DR

The applicability of S-VoCAL is demonstrated by applying a simple Retrieval-Augmented Generation (RAG) pipeline to the task of inferring character attributes, and the results suggest that the RAG pipeline reliably infers attributes such as Age or Gender, but struggles on others such as Origin or Physical Health.

Abstract

With recent advances in Text-to-Speech (TTS) systems, synthetic audiobook narration has seen increased interest, reaching unprecedented levels of naturalness. However, larger gaps remain in synthetic narration systems' ability to impersonate fictional characters, and convey complex emotions or prosody. A promising direction to enhance character identification is the assignment of plausible voices to each fictional characters in a book. This step typically requires complex inference of attributes in book-length contexts, such as a character's age, gender, origin or physical health, which in turns requires dedicated benchmark datasets to evaluate extraction systems' performances. We present S-VoCAL (Speaking Voice Character Attributes in Literature), the first dataset and evaluation framework dedicated to evaluate the inference of voice-related fictional character attributes. S-VoCAL entails 8 attributes grounded in sociophonetic studies, and 952 character-book pairs derived from Project Gutenberg. Its evaluation framework addresses the particularities of each attribute, and includes a novel similarity metric based on recent Large Language Models embeddings. We demonstrate the applicability of S-VoCAL by applying a simple Retrieval-Augmented Generation (RAG) pipeline to the task of inferring character attributes. Our results suggest that the RAG pipeline reliably infers attributes such as Age or Gender, but struggles on others such as Origin or Physical Health. The dataset and evaluation code are available at https://github.com/AbigailBerthe/S-VoCAL .

S-VoCAL: A Dataset and Evaluation Framework for Inferring Speaking Voice Character Attributes in Literature

TL;DR

The applicability of S-VoCAL is demonstrated by applying a simple Retrieval-Augmented Generation (RAG) pipeline to the task of inferring character attributes, and the results suggest that the RAG pipeline reliably infers attributes such as Age or Gender, but struggles on others such as Origin or Physical Health.

Abstract

With recent advances in Text-to-Speech (TTS) systems, synthetic audiobook narration has seen increased interest, reaching unprecedented levels of naturalness. However, larger gaps remain in synthetic narration systems' ability to impersonate fictional characters, and convey complex emotions or prosody. A promising direction to enhance character identification is the assignment of plausible voices to each fictional characters in a book. This step typically requires complex inference of attributes in book-length contexts, such as a character's age, gender, origin or physical health, which in turns requires dedicated benchmark datasets to evaluate extraction systems' performances. We present S-VoCAL (Speaking Voice Character Attributes in Literature), the first dataset and evaluation framework dedicated to evaluate the inference of voice-related fictional character attributes. S-VoCAL entails 8 attributes grounded in sociophonetic studies, and 952 character-book pairs derived from Project Gutenberg. Its evaluation framework addresses the particularities of each attribute, and includes a novel similarity metric based on recent Large Language Models embeddings. We demonstrate the applicability of S-VoCAL by applying a simple Retrieval-Augmented Generation (RAG) pipeline to the task of inferring character attributes. Our results suggest that the RAG pipeline reliably infers attributes such as Age or Gender, but struggles on others such as Origin or Physical Health. The dataset and evaluation code are available at https://github.com/AbigailBerthe/S-VoCAL .
Paper Structure (46 sections, 3 figures, 9 tables)

This paper contains 46 sections, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Voice Characteristics
  4. Character Attributes Datasets
  5. Evaluating Attribute Inference
  6. The Attribute Inference Task
  7. S-VoCAL Dataset
  8. Data Collection
  9. Age Annotation
  10. S-VoCAL Evaluation Framework
  11. Closed and Semi-Closed Class Evaluation
  12. Open-Class Evaluation
  13. Qwen3-based Metric
  14. Validation
  15. Human-Aligned Score
  16. ...and 31 more sections

Figures (3)

  • Figure 1: As attributes from S-VoCAL comes from various Gutenberg books, they are unbalanced, especially for Type that mostly contains humans and Age that mostly contains adults.
  • Figure 2: Overview of the RAG pipeline. The first step extracts attribute-specific relevant excerpts for a character. These passages are then prompted to an LLM along with an attribute prompt.
  • Figure 3: Comparison of Phi-4 and Qwen3 mHAS scores across attributes.