Learning from others' mistakes: Finetuning machine translation models with span-level error annotations

TL;DR

A simple finetuning algorithm, called Training with Annotations (TWA), is developed, to directly train machine translation models on such annotated data, which outperforms baselines such as Supervised FineTuning on sequences filtered for quality and Direct Preference Optimization on pairs constructed from the same data.

Abstract

Despite growing interest in incorporating feedback to improve language models, most efforts focus only on sequence-level annotations. In this work, we explore the potential of utilizing fine-grained span-level annotations from offline datasets to improve model quality. We develop a simple finetuning algorithm, called Training with Annotations (TWA), to directly train machine translation models on such annotated data. TWA utilizes targeted span-level error information while also flexibly learning what to penalize within a span. Moreover, TWA considers the overall trajectory of a sequence when deciding which non-error spans to utilize as positive signals. Experiments on English-German and Chinese-English machine translation show that TWA outperforms baselines such as Supervised FineTuning on sequences filtered for quality and Direct Preference Optimization on pairs constructed from the same data.

Figures (4)

• Figure 1: Overview of Training with Annotations (TWA). TWA proceeds by tokenizing the output text and its annotations. Then, a weighted span-level unlikelihood loss is applied to each error span to allow the model to learn what parts of the error span to penalize and non-error tokens following an error span are ignored as they are off-trajectory. All other tokens (i.e., non-error tokens preceding an error span) are trained with cross entropy loss.
• Figure 2: Change in the rank of each token in the vocabulary from the base model to the TWA-trained model. Dashed red lines indicate annotated errors. Red bars show a worsening in rank, while green bars indicate improvement. TWA learns diverse patterns for penalizing specific token conditionals within an error span—patterns that would be challenging to capture with heuristics.
• Figure 3: Histograms of the proportion and number of errors in the training data. Left is En-De, right is Zh-En.
• Figure 4: Histogram of COMET scores across the submissions and base model generations. Source inputs come from the training data.