Tracing Multilingual Knowledge Acquisition Dynamics in Domain Adaptation: A Case Study of English-Japanese Biomedical Adaptation

TL;DR

The study investigates how multilingual domain adaptation proceeds in large language models and introduces AdaXEval, an adaptive evaluation pipeline that builds domain-aligned QA datasets from training data to probe memorization, intralingual generalization, and cross-lingual transfer. Through a case study on English–Japanese biomedical adaptation with a 13B bilingual LLM, the authors reveal a loss-shielding mechanism that preserves correct knowledge while overfitting progresses, and show that cross-lingual transfer remains limited without domain-specific multilingual signals. The work demonstrates that perturbation analyses and multilingual data recipes (e.g., translation/romanization) are crucial to understanding and improving cross-lingual knowledge transfer, though gains are often modest relative to cost. Together, these findings offer practical guidance for designing robust multilingual domain-adaptation pipelines and point to future directions in more effective cross-lingual knowledge transfer.

Abstract

Multilingual domain adaptation (ML-DA) is widely used to learn new domain knowledge across languages into large language models (LLMs). Although many methods have been proposed to improve domain adaptation, the mechanisms of multilingual knowledge acquisition, how domain knowledge is learned within a language and transferred across languages, remain underexplored. This gap leads to suboptimal performance, particularly in low-resource settings. This work examines the learning dynamics of LLMs during ML-DA. Because prior ML-DA studies often train and evaluate on datasets with mismatched knowledge coverage, we propose AdaXEval, an adaptive evaluation method that builds multiple-choice QA datasets from the same bilingual domain corpus used for training, thereby directly studying multilingual knowledge acquisition. Through continual training of LLMs with diverse data recipes, we track how LLMs acquire domain facts and pinpoint the mechanism behind the transformation process from domain training data to knowledge. Our experiments on a 13B English-Japanese bilingual LLM reveal that cross-lingual transfer remains challenging despite a high-quality bilingual corpus. The code has been released.

Figures (14)

• Figure 1: Overview of AdaXEval, a pipeline to adaptively generate domain knowledge evaluation datasets.
• Figure 2: Dynamic knowledge acquisition evaluation after monolingual pretraining.
• Figure 3: Loss dynamics of datasets generated by AdaXEval during monolingual training.
• Figure 4: Instance state transitions before/after training.
• Figure 5: Acquired instances with different loss shapes. The line with bright circles indicates the correct answer.