Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparative Study of Multilingual Idioms and Similes in Large Language Models

Paria Khoshtab, Danial Namazifard, Mostafa Masoudi, Ali Akhgary, Samin Mahdizadeh Sani, Yadollah Yaghoobzadeh

TL;DR

This work investigates multilingual figurative language interpretation in large language models by comparing simile and idiom understanding across languages using MABL, MAPS, and newly created Persian datasets. It evaluates diverse prompting strategies (zero-shot, one-shot, Chain-of-Thought, Dialogue Simulation RiC) and translation-based prompts across closed- and open-source models, revealing language- and type-dependent performance patterns. The study finds that prompt engineering yields variable benefits depending on figurative type, language, and model, with open-source models sometimes lagging on similes in low-resource languages but performing competitively on idioms. It also highlights idiom interpretation as nearing saturation for many languages with strong models, underscoring the need for more challenging evaluations and the value of the Persian datasets for broader multilingual testing.

Abstract

This study addresses the gap in the literature concerning the comparative performance of LLMs in interpreting different types of figurative language across multiple languages. By evaluating LLMs using two multilingual datasets on simile and idiom interpretation, we explore the effectiveness of various prompt engineering strategies, including chain-of-thought, few-shot, and English translation prompts. We extend the language of these datasets to Persian as well by building two new evaluation sets. Our comprehensive assessment involves both closed-source (GPT-3.5, GPT-4o mini, Gemini 1.5), and open-source models (Llama 3.1, Qwen2), highlighting significant differences in performance across languages and figurative types. Our findings reveal that while prompt engineering methods are generally effective, their success varies by figurative type, language, and model. We also observe that open-source models struggle particularly with low-resource languages in similes. Additionally, idiom interpretation is nearing saturation for many languages, necessitating more challenging evaluations.

Comparative Study of Multilingual Idioms and Similes in Large Language Models

TL;DR

This work investigates multilingual figurative language interpretation in large language models by comparing simile and idiom understanding across languages using MABL, MAPS, and newly created Persian datasets. It evaluates diverse prompting strategies (zero-shot, one-shot, Chain-of-Thought, Dialogue Simulation RiC) and translation-based prompts across closed- and open-source models, revealing language- and type-dependent performance patterns. The study finds that prompt engineering yields variable benefits depending on figurative type, language, and model, with open-source models sometimes lagging on similes in low-resource languages but performing competitively on idioms. It also highlights idiom interpretation as nearing saturation for many languages with strong models, underscoring the need for more challenging evaluations and the value of the Persian datasets for broader multilingual testing.

Abstract

This study addresses the gap in the literature concerning the comparative performance of LLMs in interpreting different types of figurative language across multiple languages. By evaluating LLMs using two multilingual datasets on simile and idiom interpretation, we explore the effectiveness of various prompt engineering strategies, including chain-of-thought, few-shot, and English translation prompts. We extend the language of these datasets to Persian as well by building two new evaluation sets. Our comprehensive assessment involves both closed-source (GPT-3.5, GPT-4o mini, Gemini 1.5), and open-source models (Llama 3.1, Qwen2), highlighting significant differences in performance across languages and figurative types. Our findings reveal that while prompt engineering methods are generally effective, their success varies by figurative type, language, and model. We also observe that open-source models struggle particularly with low-resource languages in similes. Additionally, idiom interpretation is nearing saturation for many languages, necessitating more challenging evaluations.

Paper Structure

This paper contains 51 sections, 4 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Figurative language processing
  4. Multilingual prompt engineering
  5. Methodology
  6. Datasets
  7. MABL
  8. MAPS
  9. Persian datasets
  10. Language Categorization
  11. Models
  12. Prompting
  13. Techniques
  14. Input language
  15. Native
  16. ...and 36 more sections

Figures (4)

  • Figure 1: Our language categorization provides clearer insights for analyzing the results.
  • Figure 2: Comparing results of inputs being in native, Translated-English, and both languages in one-shot setting.
  • Figure 3: Comparing the translation methods using GPT-3.5. The average accuracy of zero-shot, one-shot, and CoT methods is reported for each translation method.
  • Figure 4: Comparing consistency (%) of closed-source models in zero-shot and CoT settings when input is in native or Translated-English. Each number represents the average on languages.