Language Models as Artificial Learners: Investigating Crosslinguistic Influence

TL;DR

The paper investigates crosslinguistic influence (CLI) using bilingual, transformer-based language models as controlled learners to isolate drivers of CLI. It manipulates L1 dominance and L2 proficiency via the L2 age of exposure and interleaved training, and evaluates L2 knowledge and priming effects across five L1s with varying syntactic distance using BLiMP and allied datasets. It provides mechanistic evidence—via LogitLens and language-neuron analyses—that L1 co-activation and L2 neuron overlap track L1-L2 distance, linking behavior to internal representations. The work demonstrates that LMs can serve as a precise computational framework for theories of human CLI, enabling principled ablations and offering insights into shared versus connected syntactic representations.

Abstract

Despite the centrality of crosslinguistic influence (CLI) to bilingualism research, human studies often yield conflicting results due to inherent experimental variance. We address these inconsistencies by using language models (LMs) as controlled statistical learners to systematically simulate CLI and isolate its underlying drivers. Specifically, we study the effect of varying the L1 language dominance and the L2 language proficiency, which we manipulate by controlling the L2 age of exposure -- defined as the training step at which the L2 is introduced. Furthermore, we investigate the impact of pretraining on L1 languages with varying syntactic distance from the L2. Using cross-linguistic priming, we analyze how activating L1 structures impacts L2 processing. Our results align with evidence from psycholinguistic studies, confirming that language dominance and proficiency are strong predictors of CLI. We further find that while priming of grammatical structures is bidirectional, the priming of ungrammatical structures is sensitive to language dominance. Finally, we provide mechanistic evidence of CLI in LMs, demonstrating that the L1 is co-activated during L2 processing and directly influences the neural circuitry recruited for the L2. More broadly, our work demonstrates that LMs can serve as a computational framework to inform theories of human CLI.

Figures (12)

• Figure 1: Setup for modeling CLI using LMs. We manipulate the age of exposure by introducing the L2 at specific training steps after pretraining on the L1. The second stage of training uses interleaved L1/L2 sequences to maintain L1 influence. We select 5 L1 languages with increasing syntactic distance from the L2.
• Figure 2: CLI effects before and after priming models with different ages of exposure. On overall, the effect of priming is consistent with syntactic distance.
• Figure 3: The surprisal difference on acceptable and unacceptable sentences of 5 L1 learner groups from FCE. The regression line shows the relationship between surprisal difference and syntactic distance. Models show a preference for text of non-native speakers that share similar L1 especially after priming.
• Figure 4: The surprisal difference of models trained and primed with romanized Greek and Korean text. Compared to non-romanized results, the effect of priming is more consistent with Latin-script results.
• Figure 5: The ratio of L1 by L2 tokens during L2 processing on BLiMP dataset. The bold lines show the average over the 5 languages.