Simultaneous Masking, Not Prompting Optimization: A Paradigm Shift in Fine-tuning LLMs for Simultaneous Translation

TL;DR

Applying the proposed SimulMask on a Falcon LLM for the IWSLT 2017 dataset, a significant translation quality improvement is observed compared to state-of-the-art prompting optimization strategies on five language pairs while reducing the computational cost.

Abstract

Large language models (LLMs) have achieved state-of-the-art performance in various language processing tasks, motivating their adoption in simultaneous translation. Current fine-tuning methods to adapt LLMs for simultaneous translation focus on prompting optimization strategies using either data augmentation or prompt structure modifications. However, these methods suffer from several issues, such as unnecessarily expanded training sets, computational inefficiency from dumping the key and value cache, increased prompt sizes, or restriction to a single decision policy. To eliminate these issues, in this work, we propose SimulMask, a new paradigm for fine-tuning LLMs for simultaneous translation. It utilizes a novel attention mask approach that models simultaneous translation during fine-tuning by masking attention for a desired decision policy. Applying the proposed SimulMask on a Falcon LLM for the IWSLT 2017 dataset, we have observed a significant translation quality improvement compared to state-of-the-art prompting optimization strategies on five language pairs while reducing the computational cost.

Figures (9)

• Figure 1: Inference Mirror Attention for matching attention during inference and fine-tuning for SimulMT.
• Figure 2: SimulMask for modeling SimulMT according to a wait-1 decision policy during fine-tuning.
• Figure 3: ALiBi biases with SimulMask.
• Figure 4: Translation quality plotted against latency for LLMs on the English-French, English-Dutch, English-Romanian, and English-German language pairs.
• Figure 5: Box plots of the computational cost of each method in GFLOPs during inference.