MMCTAgent: Multi-modal Critical Thinking Agent Framework for Complex Visual Reasoning

TL;DR

MMCTAgent addresses enduring limitations of current multi-modal large language models by integrating a human-inspired framework that combines dynamic planning, expansive tool augmentation, and a vision-based critic. The three-core components operate iteratively to decompose tasks, gather multi-modal evidence, and verify final answers, with the critic automatically deriving task-specific evaluation criteria to guide self-reflection. Across diverse image and long-form video benchmarks, MMCTAgent consistently surpasses state-of-the-art foundations and tool-based pipelines, with notable gains on image datasets and EgoSchema, and a dedicated MMCT-QA dataset demonstrating robust long-form video reasoning. The framework is modular and generic, enabling easy integration of newer models and tools, while acknowledging limitations such as potential hallucinations, external-tool dependencies, and computational overhead, which motivate future refinements in grounding and efficiency.

Abstract

Recent advancements in Multi-modal Large Language Models (MLLMs) have significantly improved their performance in tasks combining vision and language. However, challenges persist in detailed multi-modal understanding, comprehension of complex tasks, and reasoning over multi-modal information. This paper introduces MMCTAgent, a novel multi-modal critical thinking agent framework designed to address the inherent limitations of current MLLMs in complex visual reasoning tasks. Inspired by human cognitive processes and critical thinking, MMCTAgent iteratively analyzes multi-modal information, decomposes queries, plans strategies, and dynamically evolves its reasoning. Additionally, MMCTAgent incorporates critical thinking elements such as verification of final answers and self-reflection through a novel approach that defines a vision-based critic and identifies task-specific evaluation criteria, thereby enhancing its decision-making abilities. Through rigorous evaluations across various image and video understanding benchmarks, we demonstrate that MMCTAgent (with and without the critic) outperforms both foundational MLLMs and other tool-augmented pipelines.

Figures (22)

• Figure 1: MMCTAgent: Image understanding.
• Figure 2: MMCTAgent: Video understanding.
• Figure 3: MMCTAgent Overview with Planner and Reasoner, Tools, and Critic Components.
• Figure 4: Vision-based Critic overview.
• Figure 5: Image understanding and reasoning.