Lingua Custodi's participation at the WMT 2025 Terminology shared task
Jingshu Liu, Raheel Qader, Gaëtan Caillaut, Mariam Nakhlé
TL;DR
The paper addresses terminology-constrained translation for multilingual MT by fine-tuning open LLMs (Qwen3-4B and Gemma3-4B-IT) with dictionaries and instruction-generation data. It introduces a two-stage training pipeline, consisting of supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO), guided by a BLEU-based reward and a term-adherence reward defined as $R_{BLEU}(y,y^*)=\frac{BLEU(y,y^*)}{100}$ and $R_{term}(y,M)=\frac{|\\{t\in M_{target}: t \subset y\\}\|}{|M|}$. Across EN→DE/ES/RU on the WMT25 Terminology Shared Task, the approach yields substantial gains in term-match accuracy and demonstrates a favorable translation quality–constraint adherence trade-off, especially with the Gemma3_SFT_GRPO model. The work also shows data-efficient terminology transfer and suggests potential for mining parallel data for broader NMT tasks using the best-performing model.
Abstract
While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning BERT based cross-lingual sentence embeddings have yet to be explored. We systematically investigate methods for learning multilingual sentence embeddings by combining the best methods for learning monolingual and cross-lingual representations including: masked language modeling (MLM), translation language modeling (TLM), dual encoder translation ranking, and additive margin softmax. We show that introducing a pre-trained multilingual language model dramatically reduces the amount of parallel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, well above the 65.5 achieved by LASER, while still performing competitively on monolingual transfer learning benchmarks. Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh and en-de. We publicly release our best multilingual sentence embedding model for 109+ languages at https://tfhub.dev/google/LaBSE.