Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Human-in-the-Loop Approach to Improving Cross-Text Prosody Transfer

Himanshu Maurya, Atli Sigurgeirsson

TL;DR

This paper tackles cross-text prosody transfer by introducing a Human-in-the-Loop (HitL) workflow that lets humans adjust salient prosodic correlates via a web UI to better align with the target text while preserving reference prosody. Built on a Daft-Exprt baseline with FiLM-conditioned, phone-level $F_0$, energy, and duration predictors, the HitL system provides word-level and utterance-level controls that map edits through $K_w$ and $s_i$ to per-phone adjustments, with constraints such as $K_w= rac{1}{|w|} rac{}{}$ and $v'_i= rac{K'_w}{s_i}$, plus duration scaling in $[0,2]$ and ranges constrained by $ \, ext{\pm}3\sigma \, $ for $F_0$ and $ \, ext{\pm}1.5\sigma \, $ for energy. In experiments with 33 participants and 420 usable edits, edited renditions were perceived as more appropriate for the target text 57.8% of the time, though overall naturalness did not improve due to artefacts from the editing process; crucially, closeness to the reference embedding did not predict perceptual similarity. The work demonstrates that HitL can identify prosodic intents and adjust cross-text PT outputs accordingly, while highlighting limitations in naturalness and the need for more robust, lower-artifact interaction. The approach offers practical implications for making cross-text prosody transfer more controllable and robust in real-world TTS deployments.

Abstract

Text-To-Speech (TTS) prosody transfer models can generate varied prosodic renditions, for the same text, by conditioning on a reference utterance. These models are trained with a reference that is identical to the target utterance. But when the reference utterance differs from the target text, as in cross-text prosody transfer, these models struggle to separate prosody from text, resulting in reduced perceived naturalness. To address this, we propose a Human-in-the-Loop (HitL) approach. HitL users adjust salient correlates of prosody to make the prosody more appropriate for the target text, while maintaining the overall reference prosodic effect. Human adjusted renditions maintain the reference prosody while being rated as more appropriate for the target text $57.8\%$ of the time. Our analysis suggests that limited user effort suffices for these improvements, and that closeness in the latent reference space is not a reliable prosodic similarity metric for the cross-text condition.

A Human-in-the-Loop Approach to Improving Cross-Text Prosody Transfer

TL;DR

This paper tackles cross-text prosody transfer by introducing a Human-in-the-Loop (HitL) workflow that lets humans adjust salient prosodic correlates via a web UI to better align with the target text while preserving reference prosody. Built on a Daft-Exprt baseline with FiLM-conditioned, phone-level , energy, and duration predictors, the HitL system provides word-level and utterance-level controls that map edits through and to per-phone adjustments, with constraints such as and , plus duration scaling in and ranges constrained by for and for energy. In experiments with 33 participants and 420 usable edits, edited renditions were perceived as more appropriate for the target text 57.8% of the time, though overall naturalness did not improve due to artefacts from the editing process; crucially, closeness to the reference embedding did not predict perceptual similarity. The work demonstrates that HitL can identify prosodic intents and adjust cross-text PT outputs accordingly, while highlighting limitations in naturalness and the need for more robust, lower-artifact interaction. The approach offers practical implications for making cross-text prosody transfer more controllable and robust in real-world TTS deployments.

Abstract

Text-To-Speech (TTS) prosody transfer models can generate varied prosodic renditions, for the same text, by conditioning on a reference utterance. These models are trained with a reference that is identical to the target utterance. But when the reference utterance differs from the target text, as in cross-text prosody transfer, these models struggle to separate prosody from text, resulting in reduced perceived naturalness. To address this, we propose a Human-in-the-Loop (HitL) approach. HitL users adjust salient correlates of prosody to make the prosody more appropriate for the target text, while maintaining the overall reference prosodic effect. Human adjusted renditions maintain the reference prosody while being rated as more appropriate for the target text of the time. Our analysis suggests that limited user effort suffices for these improvements, and that closeness in the latent reference space is not a reliable prosodic similarity metric for the cross-text condition.
Paper Structure (12 sections, 2 equations, 4 figures, 1 table)

This paper contains 12 sections, 2 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Method
  4. Model Architecture
  5. Human-in-the-loop approach
  6. Word-level Control
  7. Utterance-level control
  8. Experiments
  9. Baseline Model
  10. Edited Speech Corpus
  11. Evaluation
  12. Conclusions

Figures (4)

  • Figure 1: Overview of the proposed method. Human-in-the-loop participants adjust word- and utterance-level $F_0$, energy and duration features to improve prosody in cross-text prosody transfer.
  • Figure 2: The distributions of phone-level control-inputs for $F_0$, energy and duration (unchanged values omitted). These results suggest that energy control was substantially less important to our HitL participants than $F_0$ and duration control.
  • Figure 3: MUSHRA-like scores of edited cross-text PT samples plotted against the cosine distance between the original reference embedding and the embedded edited cross-text PT sample. A linear regression model is fitted to this data, the shaded red area indicates the $95\%$ confidence interval.
  • Figure 4: The relationship between HitL effort and the quality of the output. Left: the line of best fit for perceived naturalness as a function of HitL effort, Right: the same but for perceived quality of prosody transfer.