PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain

TL;DR

PCA-Bench addresses the lack of integrated, error-localizing benchmarks for embodied multimodal models by evaluating perception, cognition, and action within autonomous driving, domestic robotics, and open-world games. It introduces PCA-Eval for fine-grained error localization and Embodied Instruction Evolution (EIE) to automatically synthesize diverse training data, significantly boosting open-source MLLMs and sometimes rivaling GPT-4V. The results demonstrate that strong LLMs excel at error localization, while data synthesis via EIE can bridge performance gaps, highlighting practical potential for embodied AI research and evaluation. The work also discusses alignment with human values and limitations, laying a path for more robust, transparent benchmarking in multimodal embodied decision making.

Abstract

We present PCA-Bench, a multimodal decision-making benchmark for evaluating the integrated capabilities of Multimodal Large Language Models (MLLMs). Departing from previous benchmarks focusing on simplistic tasks and individual model capability, PCA-Bench introduces three complex scenarios: autonomous driving, domestic robotics, and open-world games. Given task instructions and diverse contexts, the model is required to seamlessly integrate multiple capabilities of Perception, Cognition, and Action in a reasoning chain to make accurate decisions. Moreover, PCA-Bench features error localization capabilities, scrutinizing model inaccuracies in areas such as perception, knowledge, or reasoning. This enhances the reliability of deploying MLLMs. To balance accuracy and efficiency in evaluation, we propose PCA-Eval, an automatic evaluation protocol, and assess 10 prevalent MLLMs. The results reveal significant performance disparities between open-source models and powerful proprietary models like GPT-4 Vision. To address this, we introduce Embodied-Instruction-Evolution (EIE), an automatic framework for synthesizing instruction tuning examples in multimodal embodied environments. EIE generates 7,510 training examples in PCA-Bench and enhances the performance of open-source MLLMs, occasionally surpassing GPT-4 Vision (+3\% in decision accuracy), thereby validating the effectiveness of EIE. Our findings suggest that robust MLLMs like GPT4-Vision show promise for decision-making in embodied agents, opening new avenues for MLLM research.

Figures (14)

• Figure 1: Example of decision making with MLLMs in the Perception-Cognition-Action Chain.
• Figure 2: Instances of PCA-Bench in 3 domains.
• Figure 3: Illustration of task topology graph. Events in green represent the leaf nodes of the graph.
• Figure 4: Pipeline of the Embodied Instruction Evolution method.
• Figure 5: Performance comparsion between models' zero-shot results and models' finetuned results with the data generated by Embodied-Instruct-Evolution (EIE) method. EIE improves the performance on all domains for both LLaVA1.5-7b and Qwen-VL-Chat models. Results of LLavA1.5-13B and MMICL are in Figure \ref{['fig:sft-results-mmicl']} from appendix.