QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Models

TL;DR

This work tackles the challenge of multimodal large language models struggling with fine-grained perception and reasoning across multiple images. It proposes QG-CoC, a zero-shot prompting framework that decomposes complex questions, generates sub-question–guided captions, and integrates reasoning to answer the original query, without model fine-tuning. Through extensive experiments on open- and closed-source MLLMs across multi-image and single-image benchmarks, QG-CoC consistently surpasses existing prompting baselines and demonstrates robust improvements in challenging multi-image scenarios. The findings suggest that grounding sub-questions in targeted captions enhances cross-image understanding and provides a practical baseline for future, non-tuned multimodal reasoning systems across diverse tasks and models.

Abstract

Recently, Multimodal Large Language Models (MLLMs) encounter two key issues in multi-image contexts: (1) a lack of fine-grained perception across disparate images, and (2) a diminished capability to effectively reason over and synthesize information from multiple visual inputs. However, while various prompting methods aim to describe visual content, many existing studies focus primarily on single-image settings or specific, constrained scenarios. This leaves a critical gap in understanding and addressing how MLLMs tackle more general and complex multi-image reasoning tasks. Thus, we first extensively investigate how current prompting methods perceive fine-grained visual details and process visual information when dealing with multiple images. Our findings reveal that existing prompting methods fall short in attending to needed clues and seamlessly integrating perception and reasoning. Inspired by the findings, we propose a new zero-shot prompting method, Question-Guided Chain-of-Captions (QG-CoC), a generalized prompting approach that effectively handles problems with an arbitrary number of images. We evaluate our method on various open-source and closed-source MLLMs for multi-image and single-image benchmarks. Experimental results indicate that QG-CoC demonstrates competitive performance across tasks and exhibits robust improvements in the challenging scenarios where existing prompting methods fail.

Figures (9)

• Figure 1: An example multi-image question with different captioning settings. Text in red, green, and orange highlights our advantages. Text in blue is the correct answer. The actual prompt used for each captioning setting can be found in Appendix \ref{['sec:different_captioning_strategy']}.
• Figure 2: An example multi-image question with different prompting methods. Text in red highlights the disadvantages. Text in blue is the correct answer. The actual prompt used for each method can be found in Appendix \ref{['sec:different_prompt']}.
• Figure 3: An example multi-image question and its corresponding reasoning steps using QG-CoC. The prompts used for each step can be found Appendix \ref{['sec:qgcoc']}.
• Figure 4: Prompting methods performance by image relationships on different models (MMIU dataset).
• Figure 5: Prompting methods performance by tasks on different models. (MUIR)