Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Naturalness in LLM-Generated Utterances through Disfluency Insertion

Syed Zohaib Hassan, Pierre Lison, Pål Halvorsen

TL;DR

This work tackles the lack of natural disfluencies in LLM-produced speech by introducing a two-stage pipeline: LoRA-based fine-tuning of LLMs to insert disfluencies into fluent utterances, followed by Bark TTS to render the disfluent speech. It provides thorough automatic and human evaluations, showing that disfluency insertion increases perceived spontaneity at a modest cost to intelligibility, while revealing a tendency toward over-generation and type-specific differences in disfluency patterns. The study demonstrates measurable improvements in naturalness for conversational avatars and stress-related dialogue simulations, with detailed token-level analyses and a practical discussion of limitations and ethical considerations. The findings offer guidance on calibrating disfluency frequency and improving evaluation methodologies for future natural-speech synthesis research.

Abstract

Disfluencies are a natural feature of spontaneous human speech but are typically absent from the outputs of Large Language Models (LLMs). This absence can diminish the perceived naturalness of synthesized speech, which is an important criteria when building conversational agents that aim to mimick human behaviours. We show how the insertion of disfluencies can alleviate this shortcoming. The proposed approach involves (1) fine-tuning an LLM with Low-Rank Adaptation (LoRA) to incorporate various types of disfluencies into LLM-generated utterances and (2) synthesizing those utterances using a text-to-speech model that supports the generation of speech phenomena such as disfluencies. We evaluated the quality of the generated speech across two metrics: intelligibility and perceived spontaneity. We demonstrate through a user study that the insertion of disfluencies significantly increase the perceived spontaneity of the generated speech. This increase came, however, along with a slight reduction in intelligibility.

Enhancing Naturalness in LLM-Generated Utterances through Disfluency Insertion

TL;DR

This work tackles the lack of natural disfluencies in LLM-produced speech by introducing a two-stage pipeline: LoRA-based fine-tuning of LLMs to insert disfluencies into fluent utterances, followed by Bark TTS to render the disfluent speech. It provides thorough automatic and human evaluations, showing that disfluency insertion increases perceived spontaneity at a modest cost to intelligibility, while revealing a tendency toward over-generation and type-specific differences in disfluency patterns. The study demonstrates measurable improvements in naturalness for conversational avatars and stress-related dialogue simulations, with detailed token-level analyses and a practical discussion of limitations and ethical considerations. The findings offer guidance on calibrating disfluency frequency and improving evaluation methodologies for future natural-speech synthesis research.

Abstract

Disfluencies are a natural feature of spontaneous human speech but are typically absent from the outputs of Large Language Models (LLMs). This absence can diminish the perceived naturalness of synthesized speech, which is an important criteria when building conversational agents that aim to mimick human behaviours. We show how the insertion of disfluencies can alleviate this shortcoming. The proposed approach involves (1) fine-tuning an LLM with Low-Rank Adaptation (LoRA) to incorporate various types of disfluencies into LLM-generated utterances and (2) synthesizing those utterances using a text-to-speech model that supports the generation of speech phenomena such as disfluencies. We evaluated the quality of the generated speech across two metrics: intelligibility and perceived spontaneity. We demonstrate through a user study that the insertion of disfluencies significantly increase the perceived spontaneity of the generated speech. This increase came, however, along with a slight reduction in intelligibility.

Paper Structure

This paper contains 18 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methods
  4. Dataset
  5. Model Training
  6. Text-to-speech
  7. Evaluation
  8. Reference-based Metrics
  9. Disfluency placement
  10. Type-Specific Accuracy
  11. User Study
  12. Study Design
  13. Study Results
  14. Discussion
  15. Conclusion
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Instructions to the user study's participants
  • Figure 2: Bar-plot (mean values and 95% confidence intervals) of user ratings on intelligibility and spontaneity.