Visual Para-Thinker: Divide-and-Conquer Reasoning for Visual Comprehension

TL;DR

This paper introduces Visual Para-Thinker, representing the inaugural parallel reasoning framework for MLLMs, and develops a native multimodal implementation that facilitates high-efficiency parallel processing in the vLLM framework.

Abstract

Existing LLM test-time scaling laws emphasize the emergence of self-reflective behaviors through extended reasoning length. Nevertheless, this vertical scaling strategy often encounters plateaus in exploration as the model becomes locked into specific thinking pattern. By shifting from depth to parallelism, parallel thinking mitigates the narrowing of exploration. However, the extension of this paradigm to visual domain remains an open research question. In this paper, we first examine the role of visual partitioning in parallelized reasoning and subsequently propose two distinct strategies. Based on the above, we introduce Visual Para-Thinker, representing the inaugural parallel reasoning framework for MLLMs. To maintain path independence and promote diversity in reasoning, our approach integrates Pa-Attention alongside LPRoPE. Leveraging the vLLM framework, we have developed a native multimodal implementation that facilitates high-efficiency parallel processing. Empirical results on benchmark datasets such as V*, CountBench, RefCOCO, and HallusionBench confirm that Visual Para-Thinker successfully extends the benefits of parallel reasoning to the visual domain.

Figures (9)

• Figure 1: Schematic representations of two distinct strategies for visual partitioning. (a) illustrates Block-based partitioning, while (b) shows Scan-order partitioning.
• Figure 2: (a) illustrates the attention allocation results for Path 1 and Path 4 using the Block-based partitioning strategy during visual partitioning. The left panels present the attention maps for path 1 and path 4, while the right panels display the corresponding histograms of the spatial attention weight distributions. (b) illustrates a comparison between various test-time scaling paradigms.
• Figure 3: Visual Para-Thinker architecture. Our framework operates in two stages, namely Parallel Reasoning stage and Summary stage. In the Parallel Reasoning stage, multiple reasoning paths are generated through visual partitioning. These reasoning paths are isolated via Pa-Attention and identifiable through LPRoPE. Subsequently, in the Summary stage, the contexts from these isolated reasoning paths are integrated to derive the final output.
• Figure 4: Inference framework scheme of Visual Para-Thinker. Our inference framework is divided into three stages: Shared prefill, Parallel decoding, and Summary decoding. Shared prefill generates a common KV cache, while parallel decoding produces path-specific caches that are subsequently integrated during summary decoding.
• Figure 5: (a) depicts the attention allocation patterns observed in the counting task, while (b) compares the performance of the two visual partitioning modes across various visual tasks. (c) demonstrates the superior performance of our method.

Theorems & Definitions (9)

• Theorem 2.1
• Definition 3.1: Reasoning Path Isolation
• Theorem 3.2
• Definition 3.3: Reasoning Path Unbiasedness
• Definition 3.4: Position ID Uniformity
• Theorem 3.5
• Definition 3.6: Reasoning Path Discriminability
• Theorem 3.7
• Definition 3.8: Reasoning Path Parallelism