Code-Mixed Probes Show How Pre-Trained Models Generalise On Code-Switched Text

TL;DR

This work investigates how pre-trained multilingual language models handle code-switched text, focusing on detection, syntactic structure, and semantic representations. It introduces a novel curated Spanglish CS dataset with parallel translations into Spanish and English, supplemented by synthetic CS data, and evaluates mBERT and XLM-R across detection, syntax, and semantics using probing methods. Key findings show that PLMs can reliably detect CS text and capture cross-language syntactic and semantic information in real CS data, though performance declines with synthetic CS data, indicating a reliance on naturalistic structure for robust semantics. The contributions offer a path toward leveraging monolingual data for CS tasks and provide datasets and tools to benchmark and extend analyses to additional language pairs and model families.

Abstract

Code-switching is a prevalent linguistic phenomenon in which multilingual individuals seamlessly alternate between languages. Despite its widespread use online and recent research trends in this area, research in code-switching presents unique challenges, primarily stemming from the scarcity of labelled data and available resources. In this study we investigate how pre-trained Language Models handle code-switched text in three dimensions: a) the ability of PLMs to detect code-switched text, b) variations in the structural information that PLMs utilise to capture code-switched text, and c) the consistency of semantic information representation in code-switched text. To conduct a systematic and controlled evaluation of the language models in question, we create a novel dataset of well-formed naturalistic code-switched text along with parallel translations into the source languages. Our findings reveal that pre-trained language models are effective in generalising to code-switched text, shedding light on the abilities of these models to generalise representations to CS corpora. We release all our code and data including the novel corpus at https://github.com/francesita/code-mixed-probes.

Figures (2)

• Figure 1: Mean F-1 Scores across layers for the sentence classification task for each of the PLMs studied. In this task, probe classifiers learn to distinguish between CS and monolingual text.
• Figure 2: LID model mean F-1 Scores across layers for the probe classifiers. In this task, probe classifiers learn the LID of the tokens in CS sentences.