Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AnimeScore: A Preference-Based Dataset and Framework for Evaluating Anime-Like Speech Style

Joonyong Park, Jerry Li

Abstract

Evaluating 'anime-like' voices currently relies on costly subjective judgments, yet no standardized objective metric exists. A key challenge is that anime-likeness, unlike naturalness, lacks a shared absolute scale, making conventional Mean Opinion Score (MOS) protocols unreliable. To address this gap, we propose AnimeScore, a preference-based framework for automatic anime-likeness evaluation via pairwise ranking. We collect 15,000 pairwise judgments from 187 evaluators with free-form descriptions, and acoustic analysis reveals that perceived anime-likeness is driven by controlled resonance shaping, prosodic continuity, and deliberate articulation rather than simple heuristics such as high pitch. We show that handcrafted acoustic features reach a 69.3% AUC ceiling, while SSL-based ranking models achieve up to 90.8% AUC, providing a practical metric that can also serve as a reward signal for preference-based optimization of generative speech models.

AnimeScore: A Preference-Based Dataset and Framework for Evaluating Anime-Like Speech Style

Abstract

Evaluating 'anime-like' voices currently relies on costly subjective judgments, yet no standardized objective metric exists. A key challenge is that anime-likeness, unlike naturalness, lacks a shared absolute scale, making conventional Mean Opinion Score (MOS) protocols unreliable. To address this gap, we propose AnimeScore, a preference-based framework for automatic anime-likeness evaluation via pairwise ranking. We collect 15,000 pairwise judgments from 187 evaluators with free-form descriptions, and acoustic analysis reveals that perceived anime-likeness is driven by controlled resonance shaping, prosodic continuity, and deliberate articulation rather than simple heuristics such as high pitch. We show that handcrafted acoustic features reach a 69.3% AUC ceiling, while SSL-based ranking models achieve up to 90.8% AUC, providing a practical metric that can also serve as a reward signal for preference-based optimization of generative speech models.
Paper Structure (16 sections, 1 equation, 3 figures, 6 tables)

This paper contains 16 sections, 1 equation, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Studies on Anime Voices
  4. Speech Evaluation and Automation
  5. Data Collecting and Processing for Objective Evaluation of Anime-Like Speech
  6. Selection and Preprocessing of Speech Data for Subjective Evaluation
  7. Constructing speech set for evaluation
  8. Constructing A/B comparison pairs
  9. Subjective Evaluation Design for Labeling
  10. Experimental Results and Data Analysis
  11. Relationship between Anime-Likeness and Annotator-Identified Features
  12. Multivariate Analysis of Acoustic Features
  13. Designing Score Prediction Model for Anime-Like Speech
  14. Training Setup for Score Prediction Models
  15. Experimental Results and Data Analysis
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Corpus-wise distribution of empirical win rate for (a) train and (b) test data.
  • Figure 2: Visualizations of acoustic proxies for anime-likeness speech. (a) F1-F2 formant space distribution comparing high-win rate (top 25%) and low-win rate (bottom 25%) utterances. (b) Split violin plots of normalized spectral flux and voicing ratio distributions for the top and bottom 25% groups. (c) Overall syllable rate and articulation rate across win rate.
  • Figure 3: Architecture of an anime score prediction model