Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DEJIMA: A Novel Large-scale Japanese Dataset for Image Captioning and Visual Question Answering

Toshiki Katsube, Taiga Fukuhara, Kenichiro Ando, Yusuke Mukuta, Kohei Uehara, Tatsuya Harada

TL;DR

This work tackles the shortage of large-scale Japanese Vision-and-Language resources by introducing DEJIMA, a scalable pipeline that combines web data collection, object-grounded evidence extraction, and LLM-based refinement under grounding constraints. It produces two substantial datasets, DEJIMA-Cap and DEJIMA-VQA, each with approximately 3.88 million entries, outperforming prior Japanese V&L resources in scale while preserving linguistic naturalness and cultural relevance. Through extensive quantitative and human evaluations, DEJIMA demonstrates improved Japaneseness, grounding, and performance on Japanese multimodal benchmarks, validating the effectiveness of detection-guided refinement. The authors also provide a thorough ethics and availability plan, emphasizing commercial-permissible licensing, responsible use, and reproducibility to accelerate Japanese V&L research and applications.

Abstract

This work addresses the scarcity of high-quality, large-scale resources for Japanese Vision-and-Language (V&L) modeling. We present a scalable and reproducible pipeline that integrates large-scale web collection with rigorous filtering/deduplication, object-detection-driven evidence extraction, and Large Language Model (LLM)-based refinement under grounding constraints. Using this pipeline, we build two resources: an image-caption dataset (DEJIMA-Cap) and a VQA dataset (DEJIMA-VQA), each containing 3.88M image-text pairs, far exceeding the size of existing Japanese V&L datasets. Human evaluations demonstrate that DEJIMA achieves substantially higher Japaneseness and linguistic naturalness than datasets constructed via translation or manual annotation, while maintaining factual correctness at a level comparable to human-annotated corpora. Quantitative analyses of image feature distributions further confirm that DEJIMA broadly covers diverse visual domains characteristic of Japan, complementing its linguistic and cultural representativeness. Models trained on DEJIMA exhibit consistent improvements across multiple Japanese multimodal benchmarks, confirming that culturally grounded, large-scale resources play a key role in enhancing model performance. All data sources and modules in our pipeline are licensed for commercial use, and we publicly release the resulting dataset and metadata to encourage further research and industrial applications in Japanese V&L modeling.

DEJIMA: A Novel Large-scale Japanese Dataset for Image Captioning and Visual Question Answering

TL;DR

This work tackles the shortage of large-scale Japanese Vision-and-Language resources by introducing DEJIMA, a scalable pipeline that combines web data collection, object-grounded evidence extraction, and LLM-based refinement under grounding constraints. It produces two substantial datasets, DEJIMA-Cap and DEJIMA-VQA, each with approximately 3.88 million entries, outperforming prior Japanese V&L resources in scale while preserving linguistic naturalness and cultural relevance. Through extensive quantitative and human evaluations, DEJIMA demonstrates improved Japaneseness, grounding, and performance on Japanese multimodal benchmarks, validating the effectiveness of detection-guided refinement. The authors also provide a thorough ethics and availability plan, emphasizing commercial-permissible licensing, responsible use, and reproducibility to accelerate Japanese V&L research and applications.

Abstract

This work addresses the scarcity of high-quality, large-scale resources for Japanese Vision-and-Language (V&L) modeling. We present a scalable and reproducible pipeline that integrates large-scale web collection with rigorous filtering/deduplication, object-detection-driven evidence extraction, and Large Language Model (LLM)-based refinement under grounding constraints. Using this pipeline, we build two resources: an image-caption dataset (DEJIMA-Cap) and a VQA dataset (DEJIMA-VQA), each containing 3.88M image-text pairs, far exceeding the size of existing Japanese V&L datasets. Human evaluations demonstrate that DEJIMA achieves substantially higher Japaneseness and linguistic naturalness than datasets constructed via translation or manual annotation, while maintaining factual correctness at a level comparable to human-annotated corpora. Quantitative analyses of image feature distributions further confirm that DEJIMA broadly covers diverse visual domains characteristic of Japan, complementing its linguistic and cultural representativeness. Models trained on DEJIMA exhibit consistent improvements across multiple Japanese multimodal benchmarks, confirming that culturally grounded, large-scale resources play a key role in enhancing model performance. All data sources and modules in our pipeline are licensed for commercial use, and we publicly release the resulting dataset and metadata to encourage further research and industrial applications in Japanese V&L modeling.

Paper Structure

This paper contains 35 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Human-annotated Datasets.
  3. Web-crawled Datasets.
  4. Translated Datasets.
  5. Generated/Augmented Datasets.
  6. Dataset
  7. Creation Pipeline
  8. Stage 1: Web Collection, Filtering, and Deduplication.
  9. Stage 2: Object Detection and Tag Calibration.
  10. Stage 3: LLM-guided Generation with Grounding Constraints.
  11. Efficiency Comparison.
  12. Dataset Variants.
  13. Dataset Example.
  14. Dataset Analysis
  15. Statistical Comparison.
  16. ...and 20 more sections

Figures (4)

  • Figure 1: Pipeline: (1) web collecting with strict filtering & deduplication, (2) detection-driven evidence extraction, (3) LLM refinement with grounding/safety constraints.
  • Figure 2: Example instances from DEJIMA-Cap and DEJIMA-VQA. The figure illustrates the pipeline flow: (1) the input image is processed by the object-detection model to obtain detected objects (labels and bounding boxes); (2) the alt text and detected objects are provided as inputs to the LLM, which generates a refined caption and grounded VQA pairs in Japanese. English translations are also included in the figure.
  • Figure 3: PCA projection of CLIP image embeddings. DEJIMA covers the entire region of recruit-jp while extending into broader global contexts.
  • Figure 4: VLM Output Example for the question "What is the name of the mountain visible in the background?" We compare outputs from five pipelines against the correct answer (Mount Yotei). The All pipeline (alt-text + detection) yields correct identification with natural phrasing and explicit visual grounding, while translation-based and ablated variants either hallucinate specific locations or stay generic despite object detections. Extremely long or repetitive model outputs have been truncated for clarity.