Temporal Insight Enhancement: Mitigating Temporal Hallucination in Multimodal Large Language Models

TL;DR

The paper tackles temporal hallucination in Multimodal Large Language Models (MLLMs) when answering on-demand event questions from video inputs. It introduces a training-free correction framework that decomposes on-demand queries into iconic actions and uses external tools (CLIP, BLIP2) to locate precise timestamps, generating corrective claims to adjust the MLLM's responses. Experimental evaluation on Charades-STA shows substantial reductions in temporal hallucination and improved accuracy for both event occurrence times and event ordering. The method offers a low-cost, interpretable augmentation for temporal video QA and provides a quantitative evaluation approach for temporal capabilities in MLLMs.

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced the comprehension of multimedia content, bringing together diverse modalities such as text, images, and videos. However, a critical challenge faced by these models, especially when processing video inputs, is the occurrence of hallucinations - erroneous perceptions or interpretations, particularly at the event level. This study introduces an innovative method to address event-level hallucinations in MLLMs, focusing on specific temporal understanding in video content. Our approach leverages a novel framework that extracts and utilizes event-specific information from both the event query and the provided video to refine MLLMs' response. We propose a unique mechanism that decomposes on-demand event queries into iconic actions. Subsequently, we employ models like CLIP and BLIP2 to predict specific timestamps for event occurrences. Our evaluation, conducted using the Charades-STA dataset, demonstrates a significant reduction in temporal hallucinations and an improvement in the quality of event-related responses. This research not only provides a new perspective in addressing a critical limitation of MLLMs but also contributes a quantitatively measurable method for evaluating MLLMs in the context of temporal-related questions.

Figures (7)

• Figure 1: Examples illustrating hallucination generated by MLLMs in predicting event occurrence timestamps and sequencing.
• Figure 2: Framework overview of our temporal hallucination mitigating method.
• Figure 3: Evaluation for Task 1.
• Figure 4: Evaluation for Task 2.
• Figure 5: Illustration of event temporal hallucination correction.