Tailored Teaching with Balanced Difficulty: Elevating Reasoning in Multimodal Chain-of-Thought via Prompt Curriculum

TL;DR

This work tackles instability and suboptimal performance in multimodal chain-of-thought prompting caused by randomly or manually selected prompts. It introduces CAMS, a complexity-guided active multimodal CoT sampling framework that selects prompts by jointly leveraging model uncertainty (disagreement) and intrinsic sample complexity, producing a balanced prompt curriculum. CAMS comprises disagreement-based uncertainty, a learned complexity scorer, and a difficulty-balanced sampling strategy, validated across five benchmarks and multiple MLLMs with substantial accuracy gains and reduced sampling-induced variability. The results demonstrate that a principled curriculum design for prompts can significantly enhance multimodal reasoning in diverse visual question answering tasks, with practical implications for robust MLLM deployment.

Abstract

The effectiveness of Multimodal Chain-of-Thought (MCoT) prompting is often limited by the use of randomly or manually selected examples. These examples fail to account for both model-specific knowledge distributions and the intrinsic complexity of the tasks, resulting in suboptimal and unstable model performance. To address this, we propose a novel framework inspired by the pedagogical principle of "tailored teaching with balanced difficulty". We reframe prompt selection as a prompt curriculum design problem: constructing a well ordered set of training examples that align with the model's current capabilities. Our approach integrates two complementary signals: (1) model-perceived difficulty, quantified through prediction disagreement in an active learning setup, capturing what the model itself finds challenging; and (2) intrinsic sample complexity, which measures the inherent difficulty of each question-image pair independently of any model. By jointly analyzing these signals, we develop a difficulty-balanced sampling strategy that ensures the selected prompt examples are diverse across both dimensions. Extensive experiments conducted on five challenging benchmarks and multiple popular Multimodal Large Language Models (MLLMs) demonstrate that our method yields substantial and consistent improvements and greatly reduces performance discrepancies caused by random sampling, providing a principled and robust approach for enhancing multimodal reasoning.

Figures (6)

• Figure 1: Illustration of the motivation and key highlights of our proposed framework. (a) CoT uses random/manual prompts without analyzing model knowledge distribution or dataset features; (b) Auto-CoT uses clustering for representative prompts but ignores inter-model knowledge differences; (c) CAMS (Ours) screens optimal prompts via active learning uncertainty and complexity analysis, balancing difficulty to enhance effectiveness. We adopt the same multimodal input consisting of images and questions as in (a) and (b) for CAMS.
• Figure 2: The illustration of our CAMS framework. Dataset consists of multimodal inputs of images and text. Complexity-Based Dataset Feature Estimation calculates complexity by integrating question text and image captions to evaluate dataset characteristics. Analysis of Multimodal Model Internal Knowledge reveals the distribution of the model's internal knowledge through the uncertainty of the model's multiple predictions.
• Figure 3: Accuracy fluctuations across five tests of FS-CoT and CAMS, where Test 1–5 denote the serial numbers of each test.
• Figure 4: Comparison of accuracy between CAMS and three baseline methods in subdivided domains.
• Figure 5: Comparison of different example selection strategies. "AP_high" denotes the selection of only difficult examples (i.e., those with high uncertainty); "AP_low" denotes the selection of only easy examples (i.e., those with low uncertainty).