A Linguistically Motivated Analysis of Intonational Phrasing in Text-to-Speech Systems: Revealing Gaps in Syntactic Sensitivity

TL;DR

This study probes whether Text-to-Speech systems encode syntactic structure in their intonational phrasing, using psycholinguistic-inspired controlled stimuli that reveal how boundaries align with syntax. It compares three diverse TTS architectures and finds that simple clause boundaries are captured, while ambiguous sentences rely on explicit punctuation to place boundaries correctly. Regression analyses and targeted finetuning show punctuation cues dominate, but increasing exposure to syntactic variation can foster more distinct, structure-reflective intonation. The work highlights gaps in syntactic sensitivity within TTS and proposes data-centric strategies and evaluation resources to advance linguistically informed prosody.

Abstract

We analyze the syntactic sensitivity of Text-to-Speech (TTS) systems using methods inspired by psycholinguistic research. Specifically, we focus on the generation of intonational phrase boundaries, which can often be predicted by identifying syntactic boundaries within a sentence. We find that TTS systems struggle to accurately generate intonational phrase boundaries in sentences where syntactic boundaries are ambiguous (e.g., garden path sentences or sentences with attachment ambiguity). In these cases, systems need superficial cues such as commas to place boundaries at the correct positions. In contrast, for sentences with simpler syntactic structures, we find that systems do incorporate syntactic cues beyond surface markers. Finally, we finetune models on sentences without commas at the syntactic boundary positions, encouraging them to focus on more subtle linguistic cues. Our findings indicate that this leads to more distinct intonation patterns that better reflect the underlying structure.

Figures (7)

• Figure 1: Average durations of sentence regions in garden path sentences (top) and sentences with attachment ambiguity (bottom), generated by Parler-TTS. An intonational boundary consists of lengthening at the pre-boundary position (1), and insertion of a pause at the syntactic boundary position (2); asterisks indicate the presence of these effects. Example sentences are annotated on the x-axes; shading indicates the standard deviation across sentences.
• Figure 2: Durations of critical regions (i.e., pre-boundary word and pause at the boundary position), as generated by three TTS systems given different cues: presence or absence of a comma (light vs. dark); measurement of the pause at a syntactic or non-syntactic boundary (blue vs. orange). Black triangles are means, white lines are medians.
• Figure 3: Syntactic Sensitivity versus Mean Opinion Score across TTS systems. The F1 score represents the harmonic mean of a system's precision and recall in generating pauses at syntactic boundaries.
• Figure 4: Coefficients of LASSO-selected predictor variables for pause durations of TTS systems.
• Figure 5: Average pause duration before the function words as, for, to, with (each used in two different ways, e.g., as a preposition vs. conjunction) for three versions of Parler-TTS. Error bars indicate the standard error.