Benchmarking Multimodal CoT Reward Model Stepwise by Visual Program

TL;DR

This work tackles the challenge of producing informative, step-level reward signals for multimodal reasoning. It introduces SVIP, a visual-programming–driven pipeline that automatically generates executable code, translates code execution into interleaved CoT steps, and trains a TriAtt-CoT reward model across three dimensions: relevance, logic, and attribute. A dedicated SVIP Benchmark with SVIP-Train and SVIP-Test provides precise step-level annotations across 24 tasks and multiple modalities. Empirical results show that SVIP-Reward improves training and inference-time scaling of multimodal LLMs while reducing hallucinations and enhancing reasoning capabilities.

Abstract

Recent advancements in reward signal usage for Large Language Models (LLMs) are remarkable. However, significant challenges exist when transitioning reward signal to the multimodal domain, including labor-intensive annotations, over-reliance on one-step rewards, and inadequate evaluation. To address these issues, we propose SVIP, a novel approach to train a step-level multi-dimensional Chain-of-Thought~(CoT) reward model automatically. It generates code for solving visual tasks and transforms the analysis of code blocks into the evaluation of CoT step as training samples. Then, we train SVIP-Reward model using a multi-head attention mechanism called TriAtt-CoT. The advantages of SVIP-Reward are evident throughout the entire process of MLLM. We also introduce a benchmark for CoT reward model training and testing. Experimental results demonstrate that SVIP-Reward improves MLLM performance across training and inference-time scaling, yielding better results on benchmarks while reducing hallucinations and enhancing reasoning ability.

Figures (9)

• Figure 1: Existing reward models cannot provide multi-dimensional assessments at the step level, while SVIP evaluates each CoT step across three dimensions.
• Figure 2: The framework of SVIP: First, we generate multiple candidate code blocks for a VQA pair by the least-to-most prompt. Then, these code blocks are evaluated based on their compilability, logic, and function calls and align with the converted CoT step labels. Finally, these step-level, multi-dimensional data are used to train SVIP-Reward.
• Figure 3: Data distribution of SVIP-Train and SVIP-Test
• Figure 4: Three CoT step labels defined by SVIP and the corresponding assessment through program analyze.
• Figure 5: An illustration of reward models usage.