Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Guiding In-Context Learning of LLMs through Quality Estimation for Machine Translation

Javad Pourmostafa Roshan Sharami, Dimitar Shterionov, Pieter Spronck

TL;DR

This paper presents a novel methodology for in-context learning (ICL) that relies on a search algorithm guided by domain-specific quality estimation (QE), Leveraging the XGLM model, the methodology estimates the resulting translation quality without the need for translation references.

Abstract

The quality of output from large language models (LLMs), particularly in machine translation (MT), is closely tied to the quality of in-context examples (ICEs) provided along with the query, i.e., the text to translate. The effectiveness of these ICEs is influenced by various factors, such as the domain of the source text, the order in which the ICEs are presented, the number of these examples, and the prompt templates used. Naturally, selecting the most impactful ICEs depends on understanding how these affect the resulting translation quality, which ultimately relies on translation references or human judgment. This paper presents a novel methodology for in-context learning (ICL) that relies on a search algorithm guided by domain-specific quality estimation (QE). Leveraging the XGLM model, our methodology estimates the resulting translation quality without the need for translation references, selecting effective ICEs for MT to maximize translation quality. Our results demonstrate significant improvements over existing ICL methods and higher translation performance compared to fine-tuning a pre-trained language model (PLM), specifically mBART-50.

Guiding In-Context Learning of LLMs through Quality Estimation for Machine Translation

TL;DR

This paper presents a novel methodology for in-context learning (ICL) that relies on a search algorithm guided by domain-specific quality estimation (QE), Leveraging the XGLM model, the methodology estimates the resulting translation quality without the need for translation references.

Abstract

The quality of output from large language models (LLMs), particularly in machine translation (MT), is closely tied to the quality of in-context examples (ICEs) provided along with the query, i.e., the text to translate. The effectiveness of these ICEs is influenced by various factors, such as the domain of the source text, the order in which the ICEs are presented, the number of these examples, and the prompt templates used. Naturally, selecting the most impactful ICEs depends on understanding how these affect the resulting translation quality, which ultimately relies on translation references or human judgment. This paper presents a novel methodology for in-context learning (ICL) that relies on a search algorithm guided by domain-specific quality estimation (QE). Leveraging the XGLM model, our methodology estimates the resulting translation quality without the need for translation references, selecting effective ICEs for MT to maximize translation quality. Our results demonstrate significant improvements over existing ICL methods and higher translation performance compared to fine-tuning a pre-trained language model (PLM), specifically mBART-50.
Paper Structure (32 sections, 1 equation, 3 figures, 4 tables, 1 algorithm)

This paper contains 32 sections, 1 equation, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. ICL Using Quality Estimation for MT
  3. Unsupervised Retriever Ranking
  4. Search Algorithm Coupled with QE
  5. Experiments Setup
  6. Search Algorithm
  7. Mode 1:
  8. Mode 2:
  9. Mode 3:
  10. Quality Estimation
  11. QE data
  12. Multilingual Large Language Model
  13. Dataset and Evaluation Metrics
  14. Number of ICEs
  15. Compared Systems
  16. ...and 17 more sections

Figures (3)

  • Figure 1: Tokenized Translation Lengths comparison between R-BM25, our Mode 1, P=8, and the reference. "KS" denotes the Kolmogorov-Smirnov test, with the p-value indicating significance.
  • Figure 1: Pseudocode outlining the proposed Search Algorithm. Each phase of the methodology is annotated alongside the relevant code. Function arguments are omitted for simplicity. The first element of the returning list (temp) includes the selected prompt, its associated QE score, and the translated text.
  • Figure 2: Overview illustration showing an iteration of our proposed methodology.