An Energy-based Model for Word-level AutoCompletion in Computer-aided Translation

TL;DR

This work tackles Word-Level AutoCompletion in Computer-aided Translation by addressing limitations of a purely classification-based baseline. It introduces an energy-based model that defines the hidden representation jointly over the candidate word and source-context, enabling better use of informative source signals via attention mechanisms. To make training and inference practical, the paper combines negative sampling, a reranking inference pipeline, and CMBLM pretraining, achieving around a 6.07 percentage-point ACC gain over the previous SOTA across four language pairs and scenarios, including interactive translation. The approach demonstrates strong quantitative gains, supportive human evaluations, and meaningful improvements in interactive translation efficiency, suggesting practical benefits for CAT systems and IMT workflows.

Abstract

Word-level AutoCompletion(WLAC) is a rewarding yet challenging task in Computer-aided Translation. Existing work addresses this task through a classification model based on a neural network that maps the hidden vector of the input context into its corresponding label (i.e., the candidate target word is treated as a label). Since the context hidden vector itself does not take the label into account and it is projected to the label through a linear classifier, the model can not sufficiently leverage valuable information from the source sentence as verified in our experiments, which eventually hinders its overall performance. To alleviate this issue, this work proposes an energy-based model for WLAC, which enables the context hidden vector to capture crucial information from the source sentence. Unfortunately, training and inference suffer from efficiency and effectiveness challenges, thereby we employ three simple yet effective strategies to put our model into practice. Experiments on four standard benchmarks demonstrate that our reranking-based approach achieves substantial improvements (about 6.07%) over the previous state-of-the-art model. Further analyses show that each strategy of our approach contributes to the final performance.

Figures (8)

• Figure 1: (a) Illustration of the WLAC task in De$\Rightarrow$En. Suppose that a user has input a source sentence $\boldsymbol{x}$, partial translations ($\boldsymbol{c}_l$, $\boldsymbol{c}_r$) and is now typing some characters ($\boldsymbol{s}$). A well-trained WLAC model is expected to suggest "disease" to complete $\boldsymbol{s}$. The expected translation for $\boldsymbol{x}$ is "And disease is the common enemy of these desperate people." (b) Attention weights from "[MASK]" to words in $\boldsymbol{x}$ of the baseline method. (c) Attention weights from "disease" to words in $\boldsymbol{x}$ of our energy-based model. (Color intensity reflects the strength of attention weights.)
• Figure 2: The comparison between the network architectures for the baseline method WPM (a) and the energy-based model (b). In the baseline model, $\mathbf{h}_\textrm{[MASK]}$ does not capture the information from "disease" whereas $\mathbf{h}_\textrm{[disease]}$ does in the energy-based model. Note that "Target Encoder" is a variant of the Transformer decoder which can capture bidirectional information on the target side.
• Figure 3: The proportion of different frequency intervals on Zh$\Leftrightarrow$En and De$\Leftrightarrow$En validation datasets. Interval 1 and Interval 10 denote the most frequent interval and the most infrequent interval, respectively.
• Figure 4: Accuracy of our energy-based model and recall of ground-truth word with different $K$ on Zh$\Rightarrow$En NIST02 dataset (a) and De$\Rightarrow$En NT13 dataset (b). Experiments are conducted in the bi-context scenario.
• Figure 5: Alignment recall@$n$ on Zh$\Leftrightarrow$En NIST05 dataset with $n$ ranging from $1$ to $8$. Experiments are conducted in the bi-context scenario.