Evaluating Large Language Models on Multiword Expressions in Multilingual and Code-Switched Contexts

TL;DR

This study interrogates whether state-of-the-art LLMs can comprehend nuanced multiword expressions (MWEs) across multilingual and code-switched contexts. Using SemEval 2022 Task 2 data for English, Portuguese, and Galician, plus a novel code-switching dataset, the authors evaluate MWE detection and semantic interpretation under zero-shot and few-shot settings across four models (GPT-3.5, GPT-4, and two Meta LLaMA variants) and three tasks. They find that LLMs generally underperform compared to small, fine-tuned PLMs like xlm-roBERTa-base, with English being the most tractable and non-English results notably weaker; code-switching can improve detection but does not reliably improve semantic understanding. The work highlights persistent challenges in modeling non-compositional meaning in multilingual and mixed-language data, demonstrates the value of diverse datasets (including CS data), and argues for new approaches to better capture nuanced language beyond memorization.

Abstract

Multiword expressions, characterised by non-compositional meanings and syntactic irregularities, are an example of nuanced language. These expressions can be used literally or idiomatically, leading to significant changes in meaning. While large language models have demonstrated strong performance across many tasks, their ability to handle such linguistic subtleties remains uncertain. Therefore, this study evaluates how state-of-the-art language models process the ambiguity of potentially idiomatic multiword expressions, particularly in contexts that are less frequent, where models are less likely to rely on memorisation. By evaluating models across in Portuguese and Galician, in addition to English, and using a novel code-switched dataset and a novel task, we find that large language models, despite their strengths, struggle with nuanced language. In particular, we find that the latest models, including GPT-4, fail to outperform the xlm-roBERTa-base baselines in both detection and semantic tasks, with especially poor performance on the novel tasks we introduce, despite its similarity to existing tasks. Overall, our results demonstrate that multiword expressions, especially those which are ambiguous, continue to be a challenge to models.

Figures (3)

• Figure 1: F1 scores for each model using the original SemEval task 2 dataset. This graph shows the F1 score for the English language subset of the dataset, as well as the total macro F1 score for all languages combined in the zero-shot (zs) and few-shot settings (fs).
• Figure 2: F1 scores for each model using the CS dataset. This graph shows the F1 score for the Spanish-English (es-en) language subset of the dataset, as well as the total macro F1 score for all CS examples combined in the zero-shot (zs) and few-shot (fs) settings.
• Figure 3: The error rates of all MWEs in the test set. These are in alphabetical order per language. The MWEs in English are in the light blue background, Portuguese are in the light orange background and Galician are in a white background. The figure on the left show the errors for LLama 70B and GPT-4 and the figure on the right shows the errors for the Llama 70B and Llama 8B.