Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Speech-LLM Takes It All: A Truly Fully End-to-End Spoken Dialogue State Tracking Approach

Nizar El Ghazal, Antoine Caubrière, Valentin Vielzeuf

TL;DR

This work assesses context management in end-to-end Spoken Dialog State Tracking using Speech-LLMs, comparing multimodal, full spoken, and compressed spoken history strategies on SpokenWOZ. It introduces a three-component pipeline (speech encoder, connector, LLM) with an optional $N_{queries}$-based compression and trains in two stages: ASR pre-training to align speech embeddings and DST fine-tuning with a LoRA on the LLM. The results show that feeding the full spoken conversation yields state-of-the-art Joint Goal Accuracy for models of similar size, while attention-pooling compression provides a strong accuracy/context-size trade-off; ablations detail why full spoken context helps, especially on harder slot types. Overall, the paper demonstrates the practical viability of fully end-to-end Spoken DST with Speech-LLMs and outlines directions for scaling and more compact context handling.

Abstract

This paper presents a comparative study of context management strategies for end-to-end Spoken Dialog State Tracking using Speech-LLMs. We systematically evaluate traditional multimodal context (combining text history and spoken current turn), full spoken history, and compressed spoken history approaches. Our experiments on the SpokenWOZ corpus demonstrate that providing the full spoken conversation as input yields the highest performance among models of similar size, significantly surpassing prior methods. Furthermore, we show that attention-pooling-based compression of the spoken history offers a strong trade-off, maintaining competitive accuracy with reduced context size. Detailed analysis confirms that improvements stem from more effective context utilization.

The Speech-LLM Takes It All: A Truly Fully End-to-End Spoken Dialogue State Tracking Approach

TL;DR

This work assesses context management in end-to-end Spoken Dialog State Tracking using Speech-LLMs, comparing multimodal, full spoken, and compressed spoken history strategies on SpokenWOZ. It introduces a three-component pipeline (speech encoder, connector, LLM) with an optional -based compression and trains in two stages: ASR pre-training to align speech embeddings and DST fine-tuning with a LoRA on the LLM. The results show that feeding the full spoken conversation yields state-of-the-art Joint Goal Accuracy for models of similar size, while attention-pooling compression provides a strong accuracy/context-size trade-off; ablations detail why full spoken context helps, especially on harder slot types. Overall, the paper demonstrates the practical viability of fully end-to-end Spoken DST with Speech-LLMs and outlines directions for scaling and more compact context handling.

Abstract

This paper presents a comparative study of context management strategies for end-to-end Spoken Dialog State Tracking using Speech-LLMs. We systematically evaluate traditional multimodal context (combining text history and spoken current turn), full spoken history, and compressed spoken history approaches. Our experiments on the SpokenWOZ corpus demonstrate that providing the full spoken conversation as input yields the highest performance among models of similar size, significantly surpassing prior methods. Furthermore, we show that attention-pooling-based compression of the spoken history offers a strong trade-off, maintaining competitive accuracy with reduced context size. Detailed analysis confirms that improvements stem from more effective context utilization.

Paper Structure

This paper contains 11 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Context Management
  4. Training
  5. Results
  6. Datasets
  7. Implementation details
  8. Best Model Analysis
  9. Context Management Methods Comparison
  10. Additional Experiences and Discussion
  11. Conclusion

Figures (3)

  • Figure 1: An overview of our system. to the left, the ASR pretraining stage. To the right finetuning for dialog state tracking
  • Figure 2: Distribution of Levenshtein (fuzzy) ratios for the six most error-prone slots, with counts of insertions (orange) and deletions (red). High fuzzy ratios indicate near-correct predictions.
  • Figure 3: (a) Slot value F1 score analysis per category. (b) JGA score analysis per dialogue turn.