Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do LLMs Recognize me, When I is not me: Assessment of LLMs Understanding of Turkish Indexical Pronouns in Indexical Shift Contexts

Metehan Oğuz, Yusuf Umut Ciftci, Yavuz Faruk Bakman

TL;DR

This paper investigates how multilingual LLMs handle the Turkish indexical shift, a grammatical phenomenon where first-person indexicals can refer to either the speaker or the attitude holder depending on context. It introduces the Turkish Indexical Shift Dataset with 156 Turkish multiple-choice items, embedding verbs, and context priming, and evaluates five LLMs (GPT-4, GPT-3.5, Cohere-AYA, Trendyol-LLM, Turkcell-LLM) in a few-shot setting. The analysis shows that even strong models struggle with indexical shift, with performance strongest for finite embedded clauses and degraded for nominalized clauses, highlighting the need for language-specific grammatical understanding in LLMs. The work also provides a detailed statistical examination (LMER) of factors affecting decisions and releases dataset and code to spur further research on low-resource language challenges in LLMs.

Abstract

Large language models (LLMs) have shown impressive capabilities in tasks such as machine translation, text summarization, question answering, and solving complex mathematical problems. However, their primary training on data-rich languages like English limits their performance in low-resource languages. This study addresses this gap by focusing on the Indexical Shift problem in Turkish. The Indexical Shift problem involves resolving pronouns in indexical shift contexts, a grammatical challenge not present in high-resource languages like English. We present the first study examining indexical shift in any language, releasing a Turkish dataset specifically designed for this purpose. Our Indexical Shift Dataset consists of 156 multiple-choice questions, each annotated with necessary linguistic details, to evaluate LLMs in a few-shot setting. We evaluate recent multilingual LLMs, including GPT-4, GPT-3.5, Cohere-AYA, Trendyol-LLM, and Turkcell-LLM, using this dataset. Our analysis reveals that even advanced models like GPT-4 struggle with the grammatical nuances of indexical shift in Turkish, achieving only moderate performance. These findings underscore the need for focused research on the grammatical challenges posed by low-resource languages. We released the dataset and code \href{https://anonymous.4open.science/r/indexical_shift_llm-E1B4} {here}.

Do LLMs Recognize me, When I is not me: Assessment of LLMs Understanding of Turkish Indexical Pronouns in Indexical Shift Contexts

TL;DR

This paper investigates how multilingual LLMs handle the Turkish indexical shift, a grammatical phenomenon where first-person indexicals can refer to either the speaker or the attitude holder depending on context. It introduces the Turkish Indexical Shift Dataset with 156 Turkish multiple-choice items, embedding verbs, and context priming, and evaluates five LLMs (GPT-4, GPT-3.5, Cohere-AYA, Trendyol-LLM, Turkcell-LLM) in a few-shot setting. The analysis shows that even strong models struggle with indexical shift, with performance strongest for finite embedded clauses and degraded for nominalized clauses, highlighting the need for language-specific grammatical understanding in LLMs. The work also provides a detailed statistical examination (LMER) of factors affecting decisions and releases dataset and code to spur further research on low-resource language challenges in LLMs.

Abstract

Large language models (LLMs) have shown impressive capabilities in tasks such as machine translation, text summarization, question answering, and solving complex mathematical problems. However, their primary training on data-rich languages like English limits their performance in low-resource languages. This study addresses this gap by focusing on the Indexical Shift problem in Turkish. The Indexical Shift problem involves resolving pronouns in indexical shift contexts, a grammatical challenge not present in high-resource languages like English. We present the first study examining indexical shift in any language, releasing a Turkish dataset specifically designed for this purpose. Our Indexical Shift Dataset consists of 156 multiple-choice questions, each annotated with necessary linguistic details, to evaluate LLMs in a few-shot setting. We evaluate recent multilingual LLMs, including GPT-4, GPT-3.5, Cohere-AYA, Trendyol-LLM, and Turkcell-LLM, using this dataset. Our analysis reveals that even advanced models like GPT-4 struggle with the grammatical nuances of indexical shift in Turkish, achieving only moderate performance. These findings underscore the need for focused research on the grammatical challenges posed by low-resource languages. We released the dataset and code \href{https://anonymous.4open.science/r/indexical_shift_llm-E1B4} {here}.
Paper Structure (19 sections, 3 figures, 3 tables)

This paper contains 19 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Indexical Shift in Turkish
  3. Indexical elements.
  4. Indexical Shift.
  5. Syntactic Restrictions on Indexical Shift.
  6. Related Work
  7. Turkish Indexical Shift Dataset
  8. Experiments
  9. Experimental Design
  10. Models.
  11. Evaluation Strategy.
  12. Metrics.
  13. Main Results
  14. Which Factors Effect LLMs' Decision?
  15. GPT Family and Cohere-AYA.
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Selection analysis plot of GPT-4, GPT-3.5 and Cohere-AYA models. Their outputs are significantly influenced by the context prime, indicating the context's meaning toward either the speaker or the shifted class. No other significant factors were observed.
  • Figure 2: Selection analysis plot of Turkcell-LLM. Neither clause type nor context prime has a statistically significant effect.
  • Figure 3: Selection analysis plot of Trendyol-LLM. Neither clause type nor context prime has a statistically significant effect.