UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset

TL;DR

UltraLink addresses multilingual SFT data scarcity by separating language-specific and language-agnostic capabilities. It combines knowledge-grounded data augmentation for language-specific content with a two-stage translation workflow to prune language-agnostic data while preserving cross-lingual transfer. Trained on approximately 1 million samples across five languages, UltraLink-LM outperforms several open-source multilingual baselines on chat, math reasoning, and code generation tasks, demonstrating strong language-agnostic generalization. The approach offers an scalable, extendable path for efficient multilingual SFT in open-source LLM ecosystems, with broad practical implications for multilingual AI deployment.

Abstract

Open-source large language models (LLMs) have gained significant strength across diverse fields. Nevertheless, the majority of studies primarily concentrate on English, with only limited exploration into the realm of multilingual abilities. In this work, we therefore construct an open-source multilingual supervised fine-tuning dataset. Different from previous works that simply translate English instructions, we consider both the language-specific and language-agnostic abilities of LLMs. Firstly, we introduce a knowledge-grounded data augmentation approach to elicit more language-specific knowledge of LLMs, improving their ability to serve users from different countries. Moreover, we find modern LLMs possess strong cross-lingual transfer capabilities, thus repeatedly learning identical content in various languages is not necessary. Consequently, we can substantially prune the language-agnostic supervised fine-tuning (SFT) data without any performance degradation, making multilingual SFT more efficient. The resulting UltraLink dataset comprises approximately 1 million samples across five languages (i.e., En, Zh, Ru, Fr, Es), and the proposed data construction method can be easily extended to other languages. UltraLink-LM, which is trained on UltraLink, outperforms several representative baselines across many tasks.

Figures (13)

• Figure 1: To equip large language models with not only language-specific knowledge but also language-agnostic expertise, we construct the UltraLink dataset for multilingual SFT. For each language, UltraLink consists of four subsets, encompassing chat data with language-specific content, chat data with language-agnostic content, math data, and code data.
• Figure 2: Examples of instructions with language-specific and language-agnostic content.
• Figure 3: The proposed data augmentation method consists of two pipelines. The upper pipeline illustrates the generation of language-specific chat data. Dialogues are generated by LLMs, conditioning on language-specific knowledge extracted from Wikipedia. The language-agnostic pipeline aims to leverage existing high-quality English SFT data, using a two-stage translation mechanism to mitigate translation errors stemming from cultural differences.
• Figure 4: Structure of the prompts used for dialogue generation. The provided cultural background is enclosed within a pair of separators.
• Figure 5: Principles for generating the initial question.