On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluation

TL;DR

This work investigates temperature-constrained non-deterministic machine translation (ND-MT) and its potential to address the long-standing multi-modality challenge in MT by generating lexically diverse yet semantically equivalent translations. It systematically evaluates 22 ND-MT systems across six language directions using both lexical and semantic metrics, and introduces Group Lexical Variance Score (GLVS) to assess diversity without references. The study reveals a strong potential for ND-MT to improve candidate quality under appropriate temperature settings, but shows that conventional D-MT evaluation schemes yield unreliable rankings for ND-MT due to multi-modality and sampling effects. To tackle this, the authors propose the ExpectoSample strategy, which identifies reliable metrics and robust ND-MT systems by focusing on worst-case or stable aggregations across sampling sizes. The findings have practical implications for deploying ND-MT in real-world translation where diversity and semantic fidelity must be balanced, and they call for revised evaluation frameworks tailored to ND-MT.

Abstract

In recent years, the non-deterministic properties of language models have garnered considerable attention and have shown a significant influence on real-world applications. However, such properties remain under-explored in machine translation (MT), a complex, non-deterministic NLP task. In this study, we systematically evaluate modern MT systems and identify temperature-constrained Non-Deterministic MT (ND-MT) as a distinct phenomenon. Additionally, we demonstrate that ND-MT exhibits significant potential in addressing the multi-modality issue that has long challenged MT research and provides higher-quality candidates than Deterministic MT (D-MT) under temperature constraints. However, ND-MT introduces new challenges in evaluating system performance. Specifically, the evaluation framework designed for D-MT fails to yield consistent evaluation results when applied to ND-MT. We further investigate this emerging challenge by evaluating five state-of-the-art ND-MT systems across three open datasets using both lexical-based and semantic-based metrics at varying sampling sizes. The results reveal a Buckets effect across these systems: the lowest-quality candidate generated by ND-MT consistently determines the overall system ranking across different sampling sizes for all reasonable metrics. Furthermore, we propose the ExpectoSample strategy to automatically assess the reliability of evaluation metrics for selecting robust ND-MT.

Figures (6)

• Figure 1: Delta Mean and Std (Standard deviation) values on WMT23 EN-ZH measured by lexical and semantic metrics under temperature of 0.5 with 10 candidates, respectively. The delta value is computed with deterministic results on the same dataset under greedy decoding.
• Figure 2: Delta max values on WMT23 EN-ZH measured by lexical and semantic metrics,respectively. The delta value is computed with deterministic results on the same dataset under greedy decoding.
• Figure 3: Delta Mean values on WMT23 EN-DE and WMT23 EN-RU measured by lexical and semantic metrics, respectively.
• Figure 4: The temperature effect from qwen3-chat-8 model on WMT23 EN-ZH dataset on GLVS, BLEU bleu, ChrF++ bleu of lexical metrics and COMETDA comet20da, COMETKIWI comet22kiwi of semantic metrics.
• Figure 5: Temperature effect on Llama 2 models. While lexical metrics (left) remain stable, semantic metrics (right) show degradation at high temperatures.