Probing Multimodal Large Language Models on Cognitive Biases in Chinese Short-Video Misinformation

TL;DR

This work addresses how Multimodal Large Language Models perform on cognitive biases and misinformation in Chinese short videos. It introduces a high-quality, manually annotated dataset of 200 videos across four health domains, with fine-grained labels for experimental errors, logical fallacies, and fabricated claims, grounded in authoritative evidence. The authors evaluate eight frontier MLLMs across five modalities, showing Gemini-2.5-Pro achieves the highest belief scores in multimodal settings, while models exhibit popularity and authority biases that shape judgments. The study provides a rigorous benchmark, releases the dataset and evaluation tools, and highlights the need for bias-resilient, fact-aware multimodal reasoning in real-world misinformation contexts.

Abstract

Short-video platforms have become major channels for misinformation, where deceptive claims frequently leverage visual experiments and social cues. While Multimodal Large Language Models (MLLMs) have demonstrated impressive reasoning capabilities, their robustness against misinformation entangled with cognitive biases remains under-explored. In this paper, we introduce a comprehensive evaluation framework using a high-quality, manually annotated dataset of 200 short videos spanning four health domains. This dataset provides fine-grained annotations for three deceptive patterns, experimental errors, logical fallacies, and fabricated claims, each verified by evidence such as national standards and academic literature. We evaluate eight frontier MLLMs across five modality settings. Experimental results demonstrate that Gemini-2.5-Pro achieves the highest performance in the multimodal setting with a belief score of 71.5/100, while o3 performs the worst at 35.2. Furthermore, we investigate social cues that induce false beliefs in videos and find that models are susceptible to biases like authoritative channel IDs.

Figures (14)

• Figure 1: Overview of the data structure. Upper-left: The high-quality dataset consists of twelve fields. Upper-right: Videos from Douyin and Kuaishou are processed into visual, textual, and aural modalities, with histograms depicting token length distributions. Lower-left: Misinformation is annotated with detailed error reasons, supporting evidence, and error types. Lower-right: The dataset is categorized into four major public health domains.
• Figure 2: Belief Scores (BS) of eight models across three error types on the false video subset. Yellow "Score" lines indicate the presence of annotated error reasons in the model CoT processes, rated by Gemini-2.5-Pro.
• Figure 3: Belief Scores (BS) of four models with different levels of popularity statistics (views, likes, comments, shares) on the false video subset.
• Figure 4: The difference ($r_\text{with\_ID} - r_\text{without\_ID}$) rescaled by $\frac{100}{3}$ across four verification statuses for eight models using the Claim setting and all video data. Results on the two subsets are provided in Fig. \ref{['fig:id-change-claim-ft']} in Appendix \ref{['sec:more-results']}.
• Figure 5: Average score decrease after using channel IDs from the false set (dark blue) and the true set (light blue) on the false (F) and true (T) video subsets.