DeMPT: Decoding-enhanced Multi-phase Prompt Tuning for Making LLMs Be Better Context-aware Translators

TL;DR

This paper proposes an alternative adaptation approach, named Decoding-enhanced Multi-phase Prompt Tuning (DeMPT), to make LLMs discriminately model and utilize the inter- and intra-sentence context and more effectively adapt LLMs to context-aware NMT.

Abstract

Generally, the decoder-only large language models (LLMs) are adapted to context-aware neural machine translation (NMT) in a concatenating way, where LLMs take the concatenation of the source sentence (i.e., intra-sentence context) and the inter-sentence context as the input, and then to generate the target tokens sequentially. This adaptation strategy, i.e., concatenation mode, considers intra-sentence and inter-sentence contexts with the same priority, despite an apparent difference between the two kinds of contexts. In this paper, we propose an alternative adaptation approach, named Decoding-enhanced Multi-phase Prompt Tuning (DeMPT), to make LLMs discriminately model and utilize the inter- and intra-sentence context and more effectively adapt LLMs to context-aware NMT. First, DeMPT divides the context-aware NMT process into three separate phases. During each phase, different continuous prompts are introduced to make LLMs discriminately model various information. Second, DeMPT employs a heuristic way to further discriminately enhance the utilization of the source-side inter- and intra-sentence information at the final decoding phase. Experiments show that our approach significantly outperforms the concatenation method, and further improves the performance of LLMs in discourse modeling.

Figures (7)

• Figure 1: Comparison of different strategies for adapting LLMs to context-aware NMT. The concatenation strategy (left) treats inter-sentence and intra-sentence (referred to as the "source sentence" context in the figure) with equal importance. In contrast, our approach (right) divides context-aware NMT into three distinct phases, enabling LLMs to selectively model and leverage both inter- and intra-sentence contexts.
• Figure 2: Illustration of pipeline of multi-phase prompt tuning LLM for context-aware NMT. Red lines illustrate the procedure of enhanced decoding phase.
• Figure 3: Illustration of the procedure of our proposed decoding-enhanced approach at the $t$-th decoding step.
• Figure 4: Performance of CMT-PT and our DeMPT on ZH$\rightarrow$EN test set when using different inter-sentence context lengths.
• Figure 5: Scoring criterion for Direct Assessment. We group the score into five ranges, i.e., 0-20, 21-40, 41-60, 61-80, 81-100.