VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwan

TL;DR

VisTW addresses a critical gap by introducing the first comprehensive Vision-Language Model benchmarks for Traditional Chinese in Taiwan, comprising VisTW-MCQ (3,795 questions across 21 subjects) and VisTW-Dialogue (131 Taiwan-focused image-question pairs). It also proposes a cost-effective VLM-as-judge evaluation framework using LLMs with calibration to maintain score continuity when judge models retire, achieving a strong human-correlation of ρ = 0.8466. Empirical results reveal significant performance gaps between Traditional Chinese and Simplified Chinese VLMs, with insights on image resolution sensitivity and model scaling across 31 models. The work provides practical benchmarks and evaluation methodologies to advance culturally aware VLM assessment and informs future improvements for Traditional Chinese content in Taiwan-specific contexts.

Abstract

In this paper, we propose a comprehensive evaluation benchmark for Visual Language Models (VLM) in Traditional Chinese. Our evaluation suite, the first of its kind, contains two complementary components: (1) VisTW-MCQ, a collection of manually curated exam multi-choice questions from 21 academic subjects designed to test the broad knowledge and reasoning capabilities of VLMs; and (2) VisTW-Dialogue, an open dialogue benchmark comprising 131 image-question pairs manually created to evaluate VLMs' ability in free-form dialogue generation within Taiwanese cultural contexts. These benchmarks address a critical gap in the evaluation landscape, where existing benchmarks predominantly focus on English or Simplified Chinese, neglecting the unique linguistic and cultural aspects of Traditional Chinese used in regions like Taiwan and Hong Kong. Our analysis reveals significant performance differences across various VLMs and highlights specific challenges in processing Traditional Chinese visual content.

Figures (14)

• Figure 1: VisTW consists of two subsets: (1) MCQ - a collection of multiple-choice questions from 21 academic subjects (answer choices omitted for space); and (2) Dialogue - real-life images with corresponding questions requiring understanding of Traditional Chinese and Taiwan-specific cultural context.
• Figure 2: Effect of image resolution scaling on model performance. Left: Performance on VisTW-MCQ with both upscaled ($2\times$, $4\times$) and downscaled ($1/2$, $1/4$, $1/8$) images. Right: Performance on VisTW-Dialogue with downscaled images.
• Figure 3: Comparison of VisTW-MCQ scores with MMMU (left) and CMMU (right) on a selected subset of models of varying scales. We observe a rough correlation across the three benchmarks, though some deviations suggest differences in the specific knowledge or reasoning skills each test emphasizes.
• Figure 4: Comparison of Gemini 2.0 flash and Qwen 2.5 VL 72B score distribution rounded to the nearest integer (left) and Qwen 2.5 VL 72B scores after calibrated against Gemini 2.0 flash (right).
• Figure 5: Workflow of adding images, labeling, and evaluating