Align, Generate, Learn: A Novel Closed-Loop Framework for Cross-Lingual In-Context Learning

TL;DR

This work tackles cross-lingual in-context learning (XICL) for multilingual tasks, especially in low-resource languages, by removing dependence on external retrievers and task-specific fine-tuning. It introduces a self-supervised, closed-loop framework where a generative LLM internally selects and uses cross-language examples, trained with a retrieval-generation alignment objective, a semantic coherence loss, and a reinforcement-learning loop. The method achieves state-of-the-art performance across multilingual benchmarks, with strong gains in low-resource languages and robust generalization to unseen tasks, as validated by human evaluations. This approach offers a scalable, generalizable solution for cross-lingual reasoning and generation, potentially broadening access to high-quality language technologies for diverse language communities.

Abstract

Cross-lingual in-context learning (XICL) has emerged as a transformative paradigm for leveraging large language models (LLMs) to tackle multilingual tasks, especially for low-resource languages. However, existing approaches often rely on external retrievers or task-specific fine-tuning, limiting their scalability and generalizability. In this paper, we propose a novel self-supervised framework that harnesses the generative capabilities of LLMs to internally select and utilize task-relevant examples. Our method introduces two key objectives: a retrieval-generation alignment loss to optimize the quality of selected examples and a semantic coherence loss to ensure cross-lingual consistency. Through extensive experiments on multilingual benchmarks, our approach achieves state-of-the-art performance, significantly outperforming existing baselines. Further analysis highlights its robustness across diverse language families and its ability to generalize to unseen tasks. Human evaluations confirm the superior fluency, relevance, and semantic correctness of outputs generated by our method. This work provides a scalable, effective, and generalizable solution for cross-lingual in-context learning.