LaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Real-World Multilingual VQA

TL;DR

LaV-CoT presents a language-aware visual Chain-of-Thought framework for multilingual VQA that disentangles language and vision through a four-stage reasoning pipeline and couples it with automatic multilingual CoT data curation. It trains via a two-stage paradigm (SFT followed by GRPO) using verifiable multi-aspect rewards to enforce language consistency, structural accuracy, and semantic alignment, formalized as $R_{Multi\_Aspect} = \alpha R_{Lang} + \beta R_{Count} + \gamma R_{Answer} + \delta R_{Format}$. Empirical results on MMMB, Multilingual MMBench, and MTVQA show state-of-the-art open-source performance and competitiveness with proprietary systems, with notable gains in Arabic, Turkish, Korean, and other languages, plus strong real-world A/B test outcomes. The work demonstrates practical potential for industrial deployment in multilingual document understanding, while outlining limitations in low-resource language coverage and reasoning speed, and outlining directions for future reward modeling and broader domain applicability.

Abstract

As large vision language models (VLMs) advance, their capabilities in multilingual visual question answering (mVQA) have significantly improved. Chain-of-thought (CoT) reasoning has been proven to enhance interpretability and complex reasoning. However, most existing approaches rely primarily on textual CoT and provide limited support for multilingual multimodal reasoning, constraining their deployment in real-world applications. To address this gap, we introduce LaV-CoT, the first Language-aware Visual CoT framework with Multi-Aspect Reward Optimization. LaV-CoT incorporates an interpretable multi-stage reasoning pipeline consisting of Text Summary with Bounding Box (BBox), Language Identification, Spatial Object-level Captioning, and Step-by-step Logical Reasoning. Following this reasoning pipeline, we design an automated data curation method that generates multilingual CoT annotations through iterative generation, correction, and refinement, enabling scalable and high-quality training data. To improve reasoning and generalization, LaV-CoT adopts a two-stage training paradigm combining Supervised Fine-Tuning (SFT) with Language-aware Group Relative Policy Optimization (GRPO), guided by verifiable multi-aspect rewards including language consistency, structural accuracy, and semantic alignment. Extensive evaluations on public datasets including MMMB, Multilingual MMBench, and MTVQA show that LaV-CoT achieves up to ~9.5% accuracy improvements over open-source baselines of similar size and even surpasses models with 2$\times$ larger scales by ~2.6%. Moreover, LaV-CoT outperforms advanced proprietary models such as GPT-4o-0513 and Gemini-2.5-flash. We further conducted an online A/B test to validate our method on real-world data, highlighting its effectiveness for industrial deployment. Our code is available at this link: https://github.com/HJNVR/LaV-CoT

Figures (12)

• Figure 1: Overview of Lav-CoT: (a) Direct model answers may be incorrect and exhibit language inconsistency. (b) Incorporating CoT reasoning enhances reasoning transparency but still exhibits linguistic inconsistency. (c) Our method introduces a multi-stage reasoning pipeline, yielding accurate and consistent final answers.
• Figure 2: The framework includes an automated data generation pipeline, which leverages a multi-step reasoning process comprising (a) Text Summary with BBox, (b) Language Identification, (c) Spatial Object-level Image Captioning, and (d) Step-by-step Logical Reasoning. This reasoning pipeline first produces vanilla CoT annotations, which are then iteratively refined through rigorous verification to ensure high-quality supervision. The generated data subsequently supports a two-stage training paradigm combining SFT with GRPO. Reward computation considers language consistency, count accuracy, and the edit distance between predicted and ground-truth answers, collectively enhancing structural understanding and reinforcing robust reasoning capabilities.
• Figure 3: Comparison between Qwen2.5-VL-7B and LaV-CoT. As illustrated, Qwen2.5-VL-7B demonstrates a step-by-step reasoning process; however, it fails to perform the reasoning in the target Arabic language and produces an incorrect final deduction. In contrast, LaV-CoT effectively follows its reasoning pipeline to produce the correct final answer.
• Figure 4: Training Reward Curves
• Figure 5: Abliation Study on LaV-CoT GRPO Training.