Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learning

Menglong Cui, Jiangcun Du, Shaolin Zhu, Deyi Xiong

TL;DR

This work tackles document-level machine translation with large language models by addressing two core challenges: translation incoherence and the limited length of demonstrations in in-context learning. It proposes Context-Aware Prompting (CAP), a three-step framework that first constructs a dynamic context window using multi-level attention, then summarizes this context and retrieves $K$ similar example pairs from a datastore to serve as demonstrations, and finally performs inference with an in-context learning prompt that integrates these demonstrations. Across diverse DOCMT tasks including zero pronoun translation and literary translation, CAP consistently improves document-level BLEU, chrF2, and ZPT accuracy over baselines, and demonstrates robustness across different LLM sizes. The findings highlight the potential of dynamic context selection and datastore-guided few-shot demonstrations to enhance coherence and translation quality in document-level MT, while acknowledging limitations such as coreference handling and increased computational cost. Overall, CAP advances practical DOCMT with LLMs by effectively balancing contextual relevance and demonstration length, enabling more coherent and accurate translations in real-world settings.

Abstract

Large language models (LLMs) exhibit outstanding performance in machine translation via in-context learning. In contrast to sentence-level translation, document-level translation (DOCMT) by LLMs based on in-context learning faces two major challenges: firstly, document translations generated by LLMs are often incoherent; secondly, the length of demonstration for in-context learning is usually limited. To address these issues, we propose a Context-Aware Prompting method (CAP), which enables LLMs to generate more accurate, cohesive, and coherent translations via in-context learning. CAP takes into account multi-level attention, selects the most relevant sentences to the current one as context, and then generates a summary from these collected sentences. Subsequently, sentences most similar to the summary are retrieved from the datastore as demonstrations, which effectively guide LLMs in generating cohesive and coherent translations. We conduct extensive experiments across various DOCMT tasks, and the results demonstrate the effectiveness of our approach, particularly in zero pronoun translation (ZPT) and literary translation tasks.

Efficiently Exploring Large Language Models for Document-Level Machine Translation with In-context Learning

TL;DR

This work tackles document-level machine translation with large language models by addressing two core challenges: translation incoherence and the limited length of demonstrations in in-context learning. It proposes Context-Aware Prompting (CAP), a three-step framework that first constructs a dynamic context window using multi-level attention, then summarizes this context and retrieves similar example pairs from a datastore to serve as demonstrations, and finally performs inference with an in-context learning prompt that integrates these demonstrations. Across diverse DOCMT tasks including zero pronoun translation and literary translation, CAP consistently improves document-level BLEU, chrF2, and ZPT accuracy over baselines, and demonstrates robustness across different LLM sizes. The findings highlight the potential of dynamic context selection and datastore-guided few-shot demonstrations to enhance coherence and translation quality in document-level MT, while acknowledging limitations such as coreference handling and increased computational cost. Overall, CAP advances practical DOCMT with LLMs by effectively balancing contextual relevance and demonstration length, enabling more coherent and accurate translations in real-world settings.

Abstract

Large language models (LLMs) exhibit outstanding performance in machine translation via in-context learning. In contrast to sentence-level translation, document-level translation (DOCMT) by LLMs based on in-context learning faces two major challenges: firstly, document translations generated by LLMs are often incoherent; secondly, the length of demonstration for in-context learning is usually limited. To address these issues, we propose a Context-Aware Prompting method (CAP), which enables LLMs to generate more accurate, cohesive, and coherent translations via in-context learning. CAP takes into account multi-level attention, selects the most relevant sentences to the current one as context, and then generates a summary from these collected sentences. Subsequently, sentences most similar to the summary are retrieved from the datastore as demonstrations, which effectively guide LLMs in generating cohesive and coherent translations. We conduct extensive experiments across various DOCMT tasks, and the results demonstrate the effectiveness of our approach, particularly in zero pronoun translation (ZPT) and literary translation tasks.
Paper Structure (21 sections, 4 equations, 3 figures, 8 tables)

This paper contains 21 sections, 4 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CAP: Context-Aware Prompting
  4. Dynamic Context Window
  5. Example-Specific In-Context Demonstrations Selection
  6. Inference
  7. Experiments
  8. Setup
  9. Comparing Different Prompt Selection Strategies
  10. Comparing with Traditional NMT Models
  11. Analyzing the Effectiveness on ZPT
  12. Analyzing the Effectiveness on Literary Translation
  13. Analysis
  14. Ablation Study
  15. Visualization
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Diagram of the proposed CAP. It consists of three essential steps. (1) Dynamic Context Window: selecting context by sentence-level attention scores; (2) Summarize and Retrieve: summarizing the context and retrieving parallel sentences from the datastore, which are most similar to the summary; (3) Inference: with parallel sentences serving as demonstrations, the LLM translates the input text via in-context learning.
  • Figure 2: Average attention scores of the final layer of the Qwen-14B-Chat model. The darker the color, the higher the attention score assigned to that token. It is noteworthy that the model attends to "他" (meaning "he" in English) from the examples we extract when generating "he". Our examples successfully help the model in translating the zero pronoun "他" (meaning "he" in English).
  • Figure 3: Average attention scores of the final layer of the Qwen-14B-Chat model. The darker the color, the higher the attention score assigned to that token.