Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ThaiOCRBench: A Task-Diverse Benchmark for Vision-Language Understanding in Thai

Surapon Nonesung, Teetouch Jaknamon, Sirinya Chaiophat, Natapong Nitarach, Chanakan Wittayasakpan, Warit Sirichotedumrong, Adisai Na-Thalang, Kunat Pipatanakul

TL;DR

ThaiOCRBench fills a critical gap by providing the first comprehensive, Thai-specific benchmark for vision-language understanding on text-rich visual content. It comprises 2,808 annotated samples across 13 tasks, with design emphasizing cultural relevance and layout diversity to stress Thai-language document reasoning. Zero-shot evaluations reveal a clear performance gap between proprietary and open-source models, with fine-grained OCR and handwritten content posing the greatest challenges. An error taxonomy highlighting language bias, structural mismatch, and content inaccuracies guides future improvements in Thai VLMs. The benchmark establishes a standardized, reproducible framework to drive progress in Thai-language document understanding for low-resource scripts.

Abstract

We present ThaiOCRBench, the first comprehensive benchmark for evaluating vision-language models (VLMs) on Thai text-rich visual understanding tasks. Despite recent progress in multimodal modeling, existing benchmarks predominantly focus on high-resource languages, leaving Thai underrepresented, especially in tasks requiring document structure understanding. ThaiOCRBench addresses this gap by offering a diverse, human-annotated dataset comprising 2,808 samples across 13 task categories. We evaluate a wide range of state-of-the-art VLMs in a zero-shot setting, spanning both proprietary and open-source systems. Results show a significant performance gap, with proprietary models (e.g., Gemini 2.5 Pro) outperforming open-source counterparts. Notably, fine-grained text recognition and handwritten content extraction exhibit the steepest performance drops among open-source models. Through detailed error analysis, we identify key challenges such as language bias, structural mismatch, and hallucinated content. ThaiOCRBench provides a standardized framework for assessing VLMs in low-resource, script-complex settings, and provides actionable insights for improving Thai-language document understanding.

ThaiOCRBench: A Task-Diverse Benchmark for Vision-Language Understanding in Thai

TL;DR

ThaiOCRBench fills a critical gap by providing the first comprehensive, Thai-specific benchmark for vision-language understanding on text-rich visual content. It comprises 2,808 annotated samples across 13 tasks, with design emphasizing cultural relevance and layout diversity to stress Thai-language document reasoning. Zero-shot evaluations reveal a clear performance gap between proprietary and open-source models, with fine-grained OCR and handwritten content posing the greatest challenges. An error taxonomy highlighting language bias, structural mismatch, and content inaccuracies guides future improvements in Thai VLMs. The benchmark establishes a standardized, reproducible framework to drive progress in Thai-language document understanding for low-resource scripts.

Abstract

We present ThaiOCRBench, the first comprehensive benchmark for evaluating vision-language models (VLMs) on Thai text-rich visual understanding tasks. Despite recent progress in multimodal modeling, existing benchmarks predominantly focus on high-resource languages, leaving Thai underrepresented, especially in tasks requiring document structure understanding. ThaiOCRBench addresses this gap by offering a diverse, human-annotated dataset comprising 2,808 samples across 13 task categories. We evaluate a wide range of state-of-the-art VLMs in a zero-shot setting, spanning both proprietary and open-source systems. Results show a significant performance gap, with proprietary models (e.g., Gemini 2.5 Pro) outperforming open-source counterparts. Notably, fine-grained text recognition and handwritten content extraction exhibit the steepest performance drops among open-source models. Through detailed error analysis, we identify key challenges such as language bias, structural mismatch, and hallucinated content. ThaiOCRBench provides a standardized framework for assessing VLMs in low-resource, script-complex settings, and provides actionable insights for improving Thai-language document understanding.

Paper Structure

This paper contains 22 sections, 13 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Models with Thai Support
  4. Benchmarks for Thai Text-Rich Vision Tasks
  5. ThaiOCRBench: Dataset Construction
  6. Design Principles
  7. Task Categories Definition
  8. Data Sourcing and Annotation
  9. Dataset Statistics
  10. Experimental Design
  11. Evaluation Models
  12. Evaluation Metrics
  13. Experiment Results
  14. Zero-Shot Performance Evaluation
  15. Qualitative Error Analysis
  16. ...and 7 more sections

Figures (13)

  • Figure 1: Overview of the ThaiOCRBench data collection and annotation pipeline.
  • Figure 2: Distributional statistics of ThaiOCRBench, illustrating the coverage across task types and domain-specific categories.
  • Figure 3: Qualitative error breakdown across Top-4 models.
  • Figure 4: A screenshot of the annotation platform using Google Sheets
  • Figure 5: Comprehensive character-level statistics for both questions and answers in the ThaiOCRBench dataset.
  • ...and 8 more figures