BBA: Bi-Modal Behavioral Alignment for Reasoning with Large Vision-Language Models

TL;DR

Bi-Modal Behavioral Alignment (BBA) introduces a prompting framework that separately elicits reasoning chains from visual and DSL representations in large vision-language models, then aligns them to resolve inconsistencies and highlight critical steps. By adopting a late-fusion strategy and treating cross-modal divergences as informative signals, BBA gains substantial improvements on geometry problem-solving, chess position evaluation, and molecular property prediction with GPT-4V(ision). The method formalizes diagnostics $r_{inc}$ and alignment $\hat{r}$ to produce coherent final rationales, enabling more reliable multi-step reasoning in professional domains. These results demonstrate the value of modality-specific reasoning and cross-modal reconciliation for robust multimodal inference and explainability.

Abstract

Multimodal reasoning stands as a pivotal capability for large vision-language models (LVLMs). The integration with Domain-Specific Languages (DSL), offering precise visual representations, equips these models with the opportunity to execute more accurate reasoning in complex and professional domains. However, the vanilla Chain-of-Thought (CoT) prompting method faces challenges in effectively leveraging the unique strengths of visual and DSL representations, primarily due to their differing reasoning mechanisms. Additionally, it often falls short in addressing critical steps in multi-step reasoning tasks. To mitigate these challenges, we introduce the \underline{B}i-Modal \underline{B}ehavioral \underline{A}lignment (BBA) prompting method, designed to maximize the potential of DSL in augmenting complex multi-modal reasoning tasks. This method initiates by guiding LVLMs to create separate reasoning chains for visual and DSL representations. Subsequently, it aligns these chains by addressing any inconsistencies, thus achieving a cohesive integration of behaviors from different modalities. Our experiments demonstrate that BBA substantially improves the performance of GPT-4V(ision) on geometry problem solving ($28.34\% \to 34.22\%$), chess positional advantage prediction ($42.08\% \to 46.99\%$) and molecular property prediction ($77.47\% \to 83.52\%$).

Figures (14)

• Figure 1: Comparative analyses of different methods in problem-solving and critical step detailing. Left: Problem-solving rates across diverse problem types, where CoT$_d$ and CoT$_v$ refer to Chain-of-Thought prompting with DSL and image inputs, respectively, and CoT$_m$ represents the approach combining both inputs. Right: Average number of tokens per critical step across different methods.
• Figure 2: An instantiation of the proposed Bba method.
• Figure 3: Illustration of the prompt utilized for category annotation.
• Figure 4: Illustration of the prompt utilized for critical step identification.
• Figure 5: Illustration of the prompt utilized for categorizing each step within the generated solution.