Multilingual LLMs Inherently Reward In-Language Time-Sensitive Semantic Alignment for Low-Resource Languages

TL;DR

This work addresses the persistent gap in temporal reasoning for low-resource languages within LLMs. It shows that in-language semantic alignment yields stronger results than cross-lingual alignment in time-sensitive tasks when using X-ICL, motivating a retrieval-based cross-lingual approach. The authors introduce mTEMPREASON, a multilingual temporal reasoning benchmark, and CLiTSSA, a retriever fine-tuned with parallel cross-language data and CoSENT loss to align cross-lingual time-sensitive semantics. Empirical results across Romanian, German, and French demonstrate that CLiTSSA improves temporal reasoning performance across multiple LLMs and transfer scenarios, reducing but not eliminating the cross-linguality gap relative to monolingual settings. Overall, this work advances temporal reasoning in low-resource languages and provides a concrete, scalable path toward more inclusive multilingual LLMs.

Abstract

The unwavering disparity in labeled resources between resource-rich languages and those considered low-resource remains a significant impediment for Large Language Models (LLMs). Recent strides in cross-lingual in-context learning (X-ICL), mainly through semantically aligned examples retrieved from multilingual pre-trained transformers, have shown promise in mitigating this issue. However, our investigation reveals that LLMs intrinsically reward in-language semantically aligned cross-lingual instances over direct cross-lingual semantic alignments, with a pronounced disparity in handling time-sensitive queries in the X-ICL setup. Such queries demand sound temporal reasoning ability from LLMs, yet the advancements have predominantly focused on English. This study aims to bridge this gap by improving temporal reasoning capabilities in low-resource languages. To this end, we introduce mTEMPREASON, a temporal reasoning dataset aimed at the varied degrees of low-resource languages and propose Cross-Lingual Time-Sensitive Semantic Alignment (CLiTSSA), a novel method to improve temporal reasoning in these contexts. To facilitate this, we construct an extension of mTEMPREASON comprising pairs of parallel cross-language temporal queries along with their anticipated in-language semantic similarity scores. Our empirical evidence underscores the superior performance of CLiTSSA compared to established baselines across three languages -- Romanian, German, and French, encompassing three temporal tasks and including a diverse set of four contemporaneous LLMs. This marks a significant step forward in addressing resource disparity in the context of temporal reasoning across languages.

Figures (7)

• Figure 1: A working example of the low-resource cross-lingual prompting across three temporal tasks: L1, L2, and L3 in Romanian. The translations included in small brackets are not integral to the prompt; their purpose is solely to enhance readability.
• Figure 2: Comparison of F1 scores using box plot: a dual perspective on temporal tasks with language models and languages, pivoting on French and LLaMA3-8B, respectively.
• Figure 3: Cross–Task CLiTSSA performance across tasks with F1 scores on the French test set against the X-InSTA baseline. CLiTSSA-L$^*$ represents a retriever fine-tuned using L* training dataset where $* \in \{1,2,3\}$.
• Figure 4: A comparative analysis of F1 scores across temporal tasks in monolingual and cross-lingual scenarios utilizing LLaMA3-8B, where En$_m$ and Fr$_m$ represent monolingual settings for English and French, respectively, while Fr$_c$ is French's cross-lingual setting.
• Figure 5: Histogram-based comparison of embedding space of a retriever pre- and post CLiTSSA fine-tuning across temporal task for positive and antagonist query pairs between Romanian and English.