Preferences for Idiomatic Language are Acquired Slowly -- and Forgotten Quickly: A Case Study on Swedish
Jenny Kunz
TL;DR
This paper investigates how language models acquire idiomatic language preferences in Swedish, comparing idiomaticity with general linguistic acceptability during pretraining, continued pretraining, and instruction tuning with translated data. By introducing two novel minimal-pair benchmarks—Swedish Conventionalized Idioms and Translationese Swedish—and combining them with baseline linguistic-acceptability probes (DaLAJ and ScaLA), the authors quantify the slower, more data-hungry acquisition of idioms relative to syntax and lexical correctness. Key findings show idiomatic competence improves gradually, benefits from English pretraining and larger models, and is highly vulnerable to instruction tuning on machine-translated data, often eroding idiomatic preferences while leaving core grammar relatively intact in larger models. These results highlight important limits of translation-based instruction data for multilingual LLMs and emphasize the need for data and training strategies that preserve nuanced idiomatic behavior in Swedish, with broader implications for similar mid-resource languages. The work provides publicly documented datasets and detailed analyses that inform model development, evaluation practices, and the design of multilingual pretraining and fine-tuning pipelines that respect culturally rooted language use.
Abstract
In this study, we investigate how language models develop preferences for \textit{idiomatic} as compared to \textit{linguistically acceptable} Swedish, both during pretraining and when adapting a model from English to Swedish. To do so, we train models on Swedish from scratch and by fine-tuning English-pretrained models, probing their preferences at various checkpoints using minimal pairs that differ in linguistic acceptability or idiomaticity. For linguistic acceptability, we adapt existing benchmarks into a minimal-pair format. To assess idiomaticity, we introduce two novel datasets: one contrasting conventionalized idioms with plausible variants, and another contrasting idiomatic Swedish with Translationese. Our findings suggest that idiomatic competence emerges more slowly than other linguistic abilities, including grammatical and lexical correctness. While longer training yields diminishing returns for most tasks, idiom-related performance continues to improve, particularly in the largest model tested (8B). However, instruction tuning on data machine-translated from English -- the common approach for languages with little or no native instruction data -- causes models to rapidly lose their preference for idiomatic language.