CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation

TL;DR

CVLUE introduces a Chinese vision-language benchmark designed to evaluate VLMs in Chinese culture by curating object categories and collecting images with native-Chinese involvement. It encompasses four tasks—image-text retrieval, visual question answering, visual grounding, and visual dialogue—to probe alignment, reasoning, grounding, and interactive language use within a Chinese cultural context. Across extensive experiments with multilingual VLMs in fine-tuning and zero-shot settings, CVLUE reveals a substantial gap between English and Chinese vision-language understanding and highlights category-level Chinese-cultural knowledge deficiencies. The work also shows that fine-tuning on Chinese-culture VL data measurably enhances VLU, suggesting practical pathways to improve cross-cultural VLM performance and fostering fairer evaluation for Chinese-language multimodal models.

Abstract

Despite the rapid development of Chinese vision-language models (VLMs), most existing Chinese vision-language (VL) datasets are constructed on Western-centric images from existing English VL datasets. The cultural bias in the images makes these datasets unsuitable for evaluating VLMs in Chinese culture. To remedy this issue, we present a new Chinese Vision- Language Understanding Evaluation (CVLUE) benchmark dataset, where the selection of object categories and images is entirely driven by Chinese native speakers, ensuring that the source images are representative of Chinese culture. The benchmark contains four distinct VL tasks ranging from image-text retrieval to visual question answering, visual grounding and visual dialogue. We present a detailed statistical analysis of CVLUE and provide a baseline performance analysis with several open-source multilingual VLMs on CVLUE and its English counterparts to reveal their performance gap between English and Chinese. Our in-depth category-level analysis reveals a lack of Chinese cultural knowledge in existing VLMs. We also find that fine-tuning on Chinese culture-related VL datasets effectively enhances VLMs' understanding of Chinese culture.

Figures (27)

• Figure 1: Examples of the images and their annotation for the four tasks in CVLUE.
• Figure 2: Number of annotated objects per image for CVLUE, MS-COCO, ImageNet Detection and PASCAL VOC (average numbers are shown in parentheses).
• Figure 3: The caption length distribution of CVLUE, COCO-CN, Flickr8K-CN and Flickr30K-CN (average caption lengths are shown in parentheses).
• Figure 4: Results of QwenVL model on the CVLUE VG task, displayed by image category.
• Figure 5: Category group results of QwenVL and QwenVL-Chat on the original Chinese (zh) and translated English (en) CVLUE VG test set.