Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Making Machines Sound Sarcastic: LLM-Enhanced and Retrieval-Guided Sarcastic Speech Synthesis

Zhu Li, Yuqing Zhang, Xiyuan Gao, Shekhar Nayak, Matt Coler

TL;DR

This work tackles the challenge of sarcasm in speech synthesis by proposing a Retrieval-Augmented, LLM-enhanced TTS framework that jointly leverages semantic cues from a LoRA-tuned LLaMA 3 and prosodic exemplars retrieved via RAG. The semantic encoder captures pragmatic incongruity and discourse-level cues, while retrieved prosody provides expressive reference patterns, both integrated within a VITS backbone. Key contributions include the LoRA-based sarcasm-aware semantic encoding, retrieval-based prosodic conditioning, and a unified cross-attention mechanism that fuses phoneme, semantic, and prosodic information. Empirical results show improvements in naturalness, sarcasm expressivity, and downstream sarcasm detection, highlighting the approach's potential for more contextually appropriate sarcastic speech and its promise as data augmentation for sarcasm-related tasks.

Abstract

Sarcasm is a subtle form of non-literal language that poses significant challenges for speech synthesis due to its reliance on nuanced semantic, contextual, and prosodic cues. While existing speech synthesis research has focused primarily on broad emotional categories, sarcasm remains largely unexplored. In this paper, we propose a Large Language Model (LLM)-enhanced Retrieval-Augmented framework for sarcasm-aware speech synthesis. Our approach combines (1) semantic embeddings from a LoRA-fine-tuned LLaMA 3, which capture pragmatic incongruity and discourse-level cues of sarcasm, and (2) prosodic exemplars retrieved via a Retrieval Augmented Generation (RAG) module, which provide expressive reference patterns of sarcastic delivery. Integrated within a VITS backbone, this dual conditioning enables more natural and contextually appropriate sarcastic speech. Experiments demonstrate that our method outperforms baselines in both objective measures and subjective evaluations, yielding improvements in speech naturalness, sarcastic expressivity, and downstream sarcasm detection.

Making Machines Sound Sarcastic: LLM-Enhanced and Retrieval-Guided Sarcastic Speech Synthesis

TL;DR

This work tackles the challenge of sarcasm in speech synthesis by proposing a Retrieval-Augmented, LLM-enhanced TTS framework that jointly leverages semantic cues from a LoRA-tuned LLaMA 3 and prosodic exemplars retrieved via RAG. The semantic encoder captures pragmatic incongruity and discourse-level cues, while retrieved prosody provides expressive reference patterns, both integrated within a VITS backbone. Key contributions include the LoRA-based sarcasm-aware semantic encoding, retrieval-based prosodic conditioning, and a unified cross-attention mechanism that fuses phoneme, semantic, and prosodic information. Empirical results show improvements in naturalness, sarcasm expressivity, and downstream sarcasm detection, highlighting the approach's potential for more contextually appropriate sarcastic speech and its promise as data augmentation for sarcasm-related tasks.

Abstract

Sarcasm is a subtle form of non-literal language that poses significant challenges for speech synthesis due to its reliance on nuanced semantic, contextual, and prosodic cues. While existing speech synthesis research has focused primarily on broad emotional categories, sarcasm remains largely unexplored. In this paper, we propose a Large Language Model (LLM)-enhanced Retrieval-Augmented framework for sarcasm-aware speech synthesis. Our approach combines (1) semantic embeddings from a LoRA-fine-tuned LLaMA 3, which capture pragmatic incongruity and discourse-level cues of sarcasm, and (2) prosodic exemplars retrieved via a Retrieval Augmented Generation (RAG) module, which provide expressive reference patterns of sarcastic delivery. Integrated within a VITS backbone, this dual conditioning enables more natural and contextually appropriate sarcastic speech. Experiments demonstrate that our method outperforms baselines in both objective measures and subjective evaluations, yielding improvements in speech naturalness, sarcastic expressivity, and downstream sarcasm detection.

Paper Structure

This paper contains 15 sections, 6 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Sarcasm-Aware Semantic Encoding via LoRA
  4. RAG for Prosody Conditioning
  5. LLM-Enhanced and Retrieval-Guided Sarcastic TTS
  6. Experiments
  7. Data
  8. Experimental Setup
  9. Compared Methods
  10. Results and Discussion
  11. Sarcasm Detection Results
  12. Sarcastic Speech Synthesis Results
  13. Objective Evaluation
  14. Subjective Evaluation
  15. Conclusion

Figures (1)

  • Figure 1: Overview of the sarcasm-aware speech synthesis framework. Semantic cues from a LoRA fine-tuned LLaMA 3 and prosodic exemplars retrieved via RAG are fused within a VITS-based model to synthesize sarcastic speech.