Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Models

Dahyun Kim, Sukyung Lee, Yungi Kim, Attapol Rutherford, Chanjun Park

TL;DR

Through a thorough evaluation of various LLMs with multi-lingual capabilities, a comprehensive analysis of the proposed benchmarks and how they contribute to Thai LLM development is provided.

Abstract

The rapid advancement of large language models (LLMs) has highlighted the need for robust evaluation frameworks that assess their core capabilities, such as reasoning, knowledge, and commonsense, leading to the inception of certain widely-used benchmark suites such as the H6 benchmark. However, these benchmark suites are primarily built for the English language, and there exists a lack thereof for under-represented languages, in terms of LLM development, such as Thai. On the other hand, developing LLMs for Thai should also include enhancing the cultural understanding as well as core capabilities. To address these dual challenge in Thai LLM research, we propose two key benchmarks: Thai-H6 and Thai Cultural and Linguistic Intelligence Benchmark (ThaiCLI). Through a thorough evaluation of various LLMs with multi-lingual capabilities, we provide a comprehensive analysis of the proposed benchmarks and how they contribute to Thai LLM development. Furthermore, we will make both the datasets and evaluation code publicly available to encourage further research and development for Thai LLMs.

Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Models

TL;DR

Through a thorough evaluation of various LLMs with multi-lingual capabilities, a comprehensive analysis of the proposed benchmarks and how they contribute to Thai LLM development is provided.

Abstract

The rapid advancement of large language models (LLMs) has highlighted the need for robust evaluation frameworks that assess their core capabilities, such as reasoning, knowledge, and commonsense, leading to the inception of certain widely-used benchmark suites such as the H6 benchmark. However, these benchmark suites are primarily built for the English language, and there exists a lack thereof for under-represented languages, in terms of LLM development, such as Thai. On the other hand, developing LLMs for Thai should also include enhancing the cultural understanding as well as core capabilities. To address these dual challenge in Thai LLM research, we propose two key benchmarks: Thai-H6 and Thai Cultural and Linguistic Intelligence Benchmark (ThaiCLI). Through a thorough evaluation of various LLMs with multi-lingual capabilities, we provide a comprehensive analysis of the proposed benchmarks and how they contribute to Thai LLM development. Furthermore, we will make both the datasets and evaluation code publicly available to encourage further research and development for Thai LLMs.
Paper Structure (36 sections, 11 figures, 5 tables)

This paper contains 36 sections, 11 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Thai NLP
  4. Thai Large Language Models (LLMs)
  5. Benchmarks for Thai LLMs
  6. Thai-H6
  7. Annotation Process
  8. Dataset sizes
  9. Evaluation Methodology
  10. ThaiCLI
  11. Dataset Structure
  12. Question Distribution
  13. Factoid
  14. Instruction
  15. Answer Types
  16. ...and 21 more sections

Figures (11)

  • Figure 1: Annotation process for the Thai-H6 benchmark. Thorough human review, with emphasis on cultural and domain knowledge alignment is performed after machine translation.
  • Figure 2: Sample {Question, Chosen, Rejected} triplet from the factoid category.
  • Figure 3: Sample {Question, Chosen, Rejected} triplet from the instruction category. Note that there is a clear instruction to format the answer in two sentences.
  • Figure 4: Annotation process of the ThaiCLI benchmark. Both chosen and rejected answers undergo three rounds of human review for question-answer relevancy, alignment with Thai culture, and fluency in the Thai language.
  • Figure 5: Prompt for ThaiCLI LLM-as-a-Judge evaluation.
  • ...and 6 more figures