Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thai Winograd Schemas: A Benchmark for Thai Commonsense Reasoning

Phakphum Artkaew

TL;DR

The paper presents Thai-WS, the first Thai Winograd Schema benchmark, created by translating 285 English schemas into Thai with native-speaker validation to preserve ambiguity and cultural nuance. It evaluates several state-of-the-art LLMs (Typhoon, Claude-3, GPT-3/4, Command R+) on both English and Thai data using a prompt-based, exact-match evaluation framework, revealing a notable drop in Thai performance relative to English. Key findings show that while some models achieve strong Thai Exact scores (e.g., Claude-3-Opus) and GPT-4 remains competitive, overall Thai performance lags behind English, underscoring multilingual commonsense reasoning challenges. The work provides a rigorously developed dataset, a human baseline, and a public resource to drive future improvements in cross-lingual natural language understanding and Thai NLP benchmarking.

Abstract

Commonsense reasoning is one of the important aspect of natural language understanding, with several benchmarks developed to evaluate it. However, only a few of these benchmarks are available in languages other than English. Developing parallel benchmarks facilitates cross-lingual evaluation, enabling a better understanding of different languages. This research introduces a collection of Winograd Schemas in Thai, a novel dataset designed to evaluate commonsense reasoning capabilities in the context of the Thai language. Through a methodology involving native speakers, professional translators, and thorough validation, the schemas aim to closely reflect Thai language nuances, idioms, and cultural references while maintaining ambiguity and commonsense challenges. We evaluate the performance of popular large language models on this benchmark, revealing their strengths, limitations, and providing insights into the current state-of-the-art. Results indicate that while models like GPT-4 and Claude-3-Opus achieve high accuracy in English, their performance significantly drops in Thai, highlighting the need for further advancements in multilingual commonsense reasoning.

Thai Winograd Schemas: A Benchmark for Thai Commonsense Reasoning

TL;DR

The paper presents Thai-WS, the first Thai Winograd Schema benchmark, created by translating 285 English schemas into Thai with native-speaker validation to preserve ambiguity and cultural nuance. It evaluates several state-of-the-art LLMs (Typhoon, Claude-3, GPT-3/4, Command R+) on both English and Thai data using a prompt-based, exact-match evaluation framework, revealing a notable drop in Thai performance relative to English. Key findings show that while some models achieve strong Thai Exact scores (e.g., Claude-3-Opus) and GPT-4 remains competitive, overall Thai performance lags behind English, underscoring multilingual commonsense reasoning challenges. The work provides a rigorously developed dataset, a human baseline, and a public resource to drive future improvements in cross-lingual natural language understanding and Thai NLP benchmarking.

Abstract

Commonsense reasoning is one of the important aspect of natural language understanding, with several benchmarks developed to evaluate it. However, only a few of these benchmarks are available in languages other than English. Developing parallel benchmarks facilitates cross-lingual evaluation, enabling a better understanding of different languages. This research introduces a collection of Winograd Schemas in Thai, a novel dataset designed to evaluate commonsense reasoning capabilities in the context of the Thai language. Through a methodology involving native speakers, professional translators, and thorough validation, the schemas aim to closely reflect Thai language nuances, idioms, and cultural references while maintaining ambiguity and commonsense challenges. We evaluate the performance of popular large language models on this benchmark, revealing their strengths, limitations, and providing insights into the current state-of-the-art. Results indicate that while models like GPT-4 and Claude-3-Opus achieve high accuracy in English, their performance significantly drops in Thai, highlighting the need for further advancements in multilingual commonsense reasoning.
Paper Structure (12 sections, 2 figures, 2 tables)

This paper contains 12 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Construction
  4. Translation
  5. Validation
  6. Human baseline
  7. Experimental Setup
  8. Result and Discussion
  9. Conclusion
  10. Prompt Evaluation
  11. Models Used in the Study
  12. Consistency of errors in English LLMs

Figures (2)

  • Figure 1: Winograd Schema examples in Thai, with transliteration and corresponding English version.
  • Figure 2: An example of the prompt evaluation method, detailing the system prompt, user prompt, and expected answer. Only exact matches like "The city councilmen" were considered correct, while responses such as "The answer is The city councilmen" were not accepted, ensuring consistent and reproducible evaluations.