HLTCOE at TREC 2024 NeuCLIR Track

TL;DR

The paper investigates fine-tuning multilingual PLMs for neural CLIR through ColBERT-X models powered by PLAID, using Translate-Distill and Generate-Distill with mT5 teacher scores, plus multilingual variants for MLIR. It evaluates two reranking paradigms (mT5 cross-encoder and GPT-4o) and explores index-size reduction via token pooling, alongside two report-generation strategies (summarization-based and extractive). Key contributions include effective TD/GD pipelines, multilingual RERANKING strategies, and evidence that GPT-4o-driven re-ranking often outperforms mT5, with 450-token non-overlapping passages improving performance. The work demonstrates practical retrieval and reporting pipelines for NeuCLIR tasks, highlighting strengths across CLIR, MLIR, and technical document domains and outlining directions for future generation-based training in CLIR/MLIR.

Abstract

The HLTCOE team applied PLAID, an mT5 reranker, GPT-4 reranker, score fusion, and document translation to the TREC 2024 NeuCLIR track. For PLAID we included a variety of models and training techniques -- Translate Distill (TD), Generate Distill (GD) and multi-lingual translate-distill (MTD). TD uses scores from the mT5 model over English MS MARCO query-document pairs to learn how to score query-document pairs where the documents are translated to match the CLIR setting. GD follows TD but uses passages from the collection and queries generated by an LLM for training examples. MTD uses MS MARCO translated into multiple languages, allowing experiments on how to batch the data during training. Finally, for report generation we experimented with system combination over different runs. One family of systems used either GPT-4o or Claude-3.5-Sonnet to summarize the retrieved results from a series of decomposed sub-questions. Another system took the output from those two models and verified/combined them with Claude-3.5-Sonnet. The other family used GPT4o and GPT3.5Turbo to extract and group relevant facts from the retrieved documents based on the decomposed queries. The resulting submissions directly concatenate the grouped facts to form the report and their documents of origin as the citations. The team submitted runs to all NeuCLIR tasks: CLIR and MLIR news tasks as well as the technical documents task and the report generation task.