Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DPO-Tuned Large Language Models for Segmentation in Simultaneous Speech Translation

Zeyu Yang, Satoshi Nakamura

TL;DR

The paper tackles segmentation in simultaneous speech translation by aligning segmentation decisions with human preferences. It introduces a Direct Preference Optimization (DPO) approach to fine-tune a large language model (Qwen2.5-Omni-3B) for next-breakpoint segmentation, integrating it with SeamlessM4T v2. Evaluations on ACL 60/60 across En→De, En→Ja, and En→Zh show that the DPO-tuned LLM yields higher segmentation accuracy and consistently improves translation quality (BLEU, COMET) and latency (LAAL) versus SHAS and heuristic baselines. The work demonstrates the practical potential of preference-based tuning to create adaptive, human-aligned segmentation in real-time translation, while acknowledging computational overhead and the need for broader language coverage and human evaluation.

Abstract

Simultaneous speech translation requires accurate segmentation to balance translation quality and latency. Recent studies such as SHAS have introduced pretrained segmentation models, achieving stronger performance than heuristic rules. However, segmentation models such as SHAS, though pretrained and more robust than heuristic methods, are still constrained by supervised learning objectives and do not incorporate human preference alignment, which is crucial for natural real-time interpretation. In this work, we propose a segmentation framework based on large language models (LLMs) trained with Direct Preference Optimization (DPO). By leveraging preference alignment, our method enables LLMs to predict natural segmentation points that better meet the demands of real-time translation. We evaluate the system on the ACL 60/60 corpus across three language pairs (English-Japanese, Chinese, German), using SeamlessM4T v2 as the translation backbone. Experimental results show that our DPO-tuned LLM achieves higher segmentation accuracy than SHAS and yields consistent improvements in translation quality (BLEU, COMET) as well as latency (Average Lagging). Furthermore, our system benefits from IWSLT baselines for direct comparison. These findings highlight the potential of preference-tuned LLMs to surpass existing pretrained segmentation models and advance adaptive, human-aligned simultaneous interpretation.

DPO-Tuned Large Language Models for Segmentation in Simultaneous Speech Translation

TL;DR

The paper tackles segmentation in simultaneous speech translation by aligning segmentation decisions with human preferences. It introduces a Direct Preference Optimization (DPO) approach to fine-tune a large language model (Qwen2.5-Omni-3B) for next-breakpoint segmentation, integrating it with SeamlessM4T v2. Evaluations on ACL 60/60 across En→De, En→Ja, and En→Zh show that the DPO-tuned LLM yields higher segmentation accuracy and consistently improves translation quality (BLEU, COMET) and latency (LAAL) versus SHAS and heuristic baselines. The work demonstrates the practical potential of preference-based tuning to create adaptive, human-aligned segmentation in real-time translation, while acknowledging computational overhead and the need for broader language coverage and human evaluation.

Abstract

Simultaneous speech translation requires accurate segmentation to balance translation quality and latency. Recent studies such as SHAS have introduced pretrained segmentation models, achieving stronger performance than heuristic rules. However, segmentation models such as SHAS, though pretrained and more robust than heuristic methods, are still constrained by supervised learning objectives and do not incorporate human preference alignment, which is crucial for natural real-time interpretation. In this work, we propose a segmentation framework based on large language models (LLMs) trained with Direct Preference Optimization (DPO). By leveraging preference alignment, our method enables LLMs to predict natural segmentation points that better meet the demands of real-time translation. We evaluate the system on the ACL 60/60 corpus across three language pairs (English-Japanese, Chinese, German), using SeamlessM4T v2 as the translation backbone. Experimental results show that our DPO-tuned LLM achieves higher segmentation accuracy than SHAS and yields consistent improvements in translation quality (BLEU, COMET) as well as latency (Average Lagging). Furthermore, our system benefits from IWSLT baselines for direct comparison. These findings highlight the potential of preference-tuned LLMs to surpass existing pretrained segmentation models and advance adaptive, human-aligned simultaneous interpretation.
Paper Structure (19 sections, 1 equation, 3 figures, 1 table)

This paper contains 19 sections, 1 equation, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Task Definition
  5. Baseline: SHAS
  6. Proposed Method
  7. Base LLM for Segmentation
  8. Preference Pair Construction
  9. DPO Training
  10. Integration with SeamlessM4T v2
  11. Experiments
  12. Datasets
  13. Baselines
  14. Evaluation Metrics
  15. Implementation Details
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Overview of our proposed simultaneous speech translation (SimulST) pipeline. Source audio is processed by our proposed segmentation module (Qwen2.5-Omni-3B fine-tuned with DPO), which predicts natural breakpoints and generates chunks. These chunks are then translated by SeamlessM4T v2, producing the final translation output. The highlighted segmentation module represents our main contribution.
  • Figure 2: Latency–quality tradeoff curves on ACL 60/60 (En$\rightarrow$Zh). Our method consistently outperforms baselines across latency ranges.
  • Figure 3: Latency–quality tradeoff curves for En$\rightarrow$De. Ours achieves higher BLEU at comparable latency.