Evaluating Multimodal Large Language Models on Vertically Written Japanese Text

TL;DR

This work targets the OCR-readability of vertically written Japanese text by Multimodal LLMs. It introduces JSSODa, a large synthetic dataset with horizontal and vertical layouts, and VJRODa, a real-world vertical-Japanese OCR dataset, enabling both training and evaluation. Experiments show current MLLMs underperform on vertical writing relative to horizontal, though fine-tuning on JSSODa can boost vertical-reading ability for some models, while real-world performance still lags, indicating a need for real-world OCR data. The publicly released datasets and scripts provide a path toward improved vertical-text understanding in multilingual document contexts.

Abstract

Multimodal Large Language Models (MLLMs) have seen rapid advances in recent years and are now being applied to visual document understanding tasks. They are expected to process a wide range of document images across languages, including Japanese. Understanding documents from images requires models to read what are written in them. Since some Japanese documents are written vertically, support for vertical writing is essential. However, research specifically focused on vertically written Japanese text remains limited. In this study, we evaluate the reading capability of existing MLLMs on vertically written Japanese text. First, we generate a synthetic Japanese OCR dataset by rendering Japanese texts into images, and use it for both model fine-tuning and evaluation. This dataset includes Japanese text in both horizontal and vertical writing. We also create an evaluation dataset sourced from the real-world document images containing vertically written Japanese text. Using these datasets, we demonstrate that the existing MLLMs perform worse on vertically written Japanese text than on horizontally written Japanese text. Furthermore, we show that training MLLMs on our synthesized Japanese OCR dataset results in improving the performance of models that previously could not handle vertical writing. The datasets and code are publicly available https://github.com/llm-jp/eval_vertical_ja.

Figures (15)

• Figure 1: Examples of reading order for Japanese documents (Top: Horizontal writing, Bottom: Vertical writing). The blue numbers attached to each line indicate the order in which the text on that line should be read. In horizontal writing, similar to English documents, characters in each line are read from left to right, and lines are read from top to bottom. In vertical writing, characters in each line are read from top to bottom, and lines are read from right to left.
• Figure 2: vertical, 2-columns
• Figure 3: horizontal, 3-columns
• Figure 5: An example image from VJRODa. The blue number above each line indicates the order in which the text on that line should be read. Characters in each line are read from top to bottom, and lines are read from right to left. Each column is read from top to bottom. (https://warp.ndl.go.jp/info:ndljp/pid/11712522/www.vill.kariwa.niigata.jp/open/info/000000001_0000000609.pdf, page 8, Personal information has been masked.)
• Figure 6: The example outputs on JSSODa (vertical) generated by Qwen2.5-VL-7B and its fine-tuned (+FT) variant. The red arrow ($\rightarrow$) represents the reading order of Qwen2.5-VL-7B, and the blue arrow ($\rightarrow$) represents the reading order of the fine-tuned model.