Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation

Peerat Limkonchotiwat, Kanruethai Masuk, Surapon Nonesung, Chalermpun Mai-On, Sarana Nutanong, Wuttikorn Ponwitayarat, Potsawee Manakul

TL;DR

This work exposes a significant gap in LLM performance when processing Thai local dialects (Isan, Lanna, Dambro) across five NLP tasks. It introduces a dedicated benchmark and a human-centered evaluation guideline, plus a novel local-dialect fluency metric to complement traditional automatic metrics. The results show pronounced declines in dialect understanding and generation for most models, with proprietary systems like GPT-4o and Gemini2 showing some dialect fluency, while open models struggle to produce dialect-specific output. Collectively, the benchmark and metric provide a framework to assess and guide future improvements in dialect-aware LLMs, highlighting the need for more robust multilingual and dialect-aware capabilities in LLM research and deployment.

Abstract

Large language models show promising results in various NLP tasks. Despite these successes, the robustness and consistency of LLMs in underrepresented languages remain largely unexplored, especially concerning local dialects. Existing benchmarks also focus on main dialects, neglecting LLMs' ability on local dialect texts. In this paper, we introduce a Thai local dialect benchmark covering Northern (Lanna), Northeastern (Isan), and Southern (Dambro) Thai, evaluating LLMs on five NLP tasks: summarization, question answering, translation, conversation, and food-related tasks. Furthermore, we propose a human evaluation guideline and metric for Thai local dialects to assess generation fluency and dialect-specific accuracy. Results show that LLM performance declines significantly in local Thai dialects compared to standard Thai, with only proprietary models like GPT-4o and Gemini2 demonstrating some fluency

Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation

TL;DR

This work exposes a significant gap in LLM performance when processing Thai local dialects (Isan, Lanna, Dambro) across five NLP tasks. It introduces a dedicated benchmark and a human-centered evaluation guideline, plus a novel local-dialect fluency metric to complement traditional automatic metrics. The results show pronounced declines in dialect understanding and generation for most models, with proprietary systems like GPT-4o and Gemini2 showing some dialect fluency, while open models struggle to produce dialect-specific output. Collectively, the benchmark and metric provide a framework to assess and guide future improvements in dialect-aware LLMs, highlighting the need for more robust multilingual and dialect-aware capabilities in LLM research and deployment.

Abstract

Large language models show promising results in various NLP tasks. Despite these successes, the robustness and consistency of LLMs in underrepresented languages remain largely unexplored, especially concerning local dialects. Existing benchmarks also focus on main dialects, neglecting LLMs' ability on local dialect texts. In this paper, we introduce a Thai local dialect benchmark covering Northern (Lanna), Northeastern (Isan), and Southern (Dambro) Thai, evaluating LLMs on five NLP tasks: summarization, question answering, translation, conversation, and food-related tasks. Furthermore, we propose a human evaluation guideline and metric for Thai local dialects to assess generation fluency and dialect-specific accuracy. Results show that LLM performance declines significantly in local Thai dialects compared to standard Thai, with only proprietary models like GPT-4o and Gemini2 demonstrating some fluency

Paper Structure

This paper contains 21 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Benchmark and Metric
  3. Benchmark Formulation
  4. Human Evaluation for Local Dialects
  5. Experimental Setups and Results
  6. Setup
  7. Traditional Metric Results
  8. Human Evaluation
  9. Our Local Dialect Metric
  10. Conclusion
  11. Data Statistics and Annotators
  12. Data Statistics
  13. Word Overlap
  14. Annotators
  15. Annotator Agreement
  16. ...and 6 more sections

Figures (4)

  • Figure 1: (a) We showed some unique characteristics of Thai local dialects compared to Central Thai: 1. shared tokens in local dialects; 2. unique words for each local dialect; and 3. WH- question tokens that are different in each local dialect. (b) Illustration of our local dialect evaluation metric. The example demonstrates the assessment of generation and fluency, highlighting variations in spelling and pronunciation among different Thai dialects.
  • Figure 2: Food and Conversation topics. Note that these food names are very local in each part of Thailand.
  • Figure 3: The Thai prompts that we used in our experiments with the translation version.
  • Figure 4: Example from the translation task.