Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generating Diverse Translation with Perturbed kNN-MT

Yuto Nishida, Makoto Morishita, Hidetaka Kamigaito, Taro Watanabe

TL;DR

The paper tackles limited translation diversity by addressing overcorrection in NMT with perturbed kNN-MT, introducing noised-, randomized-, and uniquify-kNN variants integrated with diversified decoding. By perturbing the kNN retrieval, the method expands the search space and allows more diverse target tokens to be considered, while maintaining fluency and largely preserving translation quality. The approach leverages datastore-based retrieval, interpolation with MT probabilities, and both static/adaptive noise or random sampling to control diversity through perturbation magnitude. Empirical results across multiple domain adaptation and general-domain language pairs demonstrate substantial gains in diversity (DP) with manageable quality trade-offs, and show that Randomized-kNN often provides the best practical balance without extra cost. The work highlights the potential of combining kNN-MT with diversification to enable more controllable and diverse translation generation in real-world applications, while noting limitations such as latency, memory usage, possible hallucinations, and retrieval challenges that warrant future research.

Abstract

Generating multiple translation candidates would enable users to choose the one that satisfies their needs. Although there has been work on diversified generation, there exists room for improving the diversity mainly because the previous methods do not address the overcorrection problem -- the model underestimates a prediction that is largely different from the training data, even if that prediction is likely. This paper proposes methods that generate more diverse translations by introducing perturbed k-nearest neighbor machine translation (kNN-MT). Our methods expand the search space of kNN-MT and help incorporate diverse words into candidates by addressing the overcorrection problem. Our experiments show that the proposed methods drastically improve candidate diversity and control the degree of diversity by tuning the perturbation's magnitude.

Generating Diverse Translation with Perturbed kNN-MT

TL;DR

The paper tackles limited translation diversity by addressing overcorrection in NMT with perturbed kNN-MT, introducing noised-, randomized-, and uniquify-kNN variants integrated with diversified decoding. By perturbing the kNN retrieval, the method expands the search space and allows more diverse target tokens to be considered, while maintaining fluency and largely preserving translation quality. The approach leverages datastore-based retrieval, interpolation with MT probabilities, and both static/adaptive noise or random sampling to control diversity through perturbation magnitude. Empirical results across multiple domain adaptation and general-domain language pairs demonstrate substantial gains in diversity (DP) with manageable quality trade-offs, and show that Randomized-kNN often provides the best practical balance without extra cost. The work highlights the potential of combining kNN-MT with diversification to enable more controllable and diverse translation generation in real-world applications, while noting limitations such as latency, memory usage, possible hallucinations, and retrieval challenges that warrant future research.

Abstract

Generating multiple translation candidates would enable users to choose the one that satisfies their needs. Although there has been work on diversified generation, there exists room for improving the diversity mainly because the previous methods do not address the overcorrection problem -- the model underestimates a prediction that is largely different from the training data, even if that prediction is likely. This paper proposes methods that generate more diverse translations by introducing perturbed k-nearest neighbor machine translation (kNN-MT). Our methods expand the search space of kNN-MT and help incorporate diverse words into candidates by addressing the overcorrection problem. Our experiments show that the proposed methods drastically improve candidate diversity and control the degree of diversity by tuning the perturbation's magnitude.
Paper Structure (53 sections, 8 equations, 5 figures, 23 tables)

This paper contains 53 sections, 8 equations, 5 figures, 23 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Diverse Text Generation
  4. Datastore creation
  5. Generation
  6. Overcorrection
  7. Static noise
  8. Adaptive noise
  9. Experiments
  10. Experimental Settings
  11. Dataset
  12. Models
  13. Baseline
  14. $k$NN-MT
  15. Diversified decoding
  16. ...and 38 more sections

Figures (5)

  • Figure 1: Overview of our proposed method: See § \ref{['sec:proposed_method']} for details. Green points represent target tokens in datastore. Blue points represent query vectors, and surrounding circles denote retrieved neighbors. (1) Noised-$k$NN adds a noise vector to the query, changing the retrieved tokens. (2) Randomized-$k$NN initially retrieves more neighbors and randomly selects $k$-neighbors. (3) Uniquify-$k$NN only considers unique target tokens from retrieved neighbors. In this figure, number of neighbors $k$ per query is set to $3$, and hyperparameter $h$ of (2) Randomized-$k$NN is set to $2.0$.
  • Figure 2: Relationship between translation quality (BLEU@20) and diversity (DP) in De-En IT domain: Top-right is most desirable.
  • Figure 3: Example 20-best lists using DBS-based methods: In upper example, DBS+$k$NN-MT+Randomize successfully diversified list by adding a likely word, COVID-19, which did not appear in DBS+$k$NN-MT. In lower example, DBS+$k$NN-MT+Randomize introduced an unlikely word, winter, which did not appear in DBS.
  • Figure 4: Relationship among perturbation's magnitudes or temperature and DP/BLEU on the De-En IT domain
  • Figure 5: Full example 20-best lists using DBS-based methods