Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs

TL;DR

Hallu-PI addresses the gap in assessing hallucination in multimodal LLMs when inputs are perturbed. It introduces a seven-perturbation benchmark with 1,260 perturbed images across 11 object types, rich per-image annotations, and templates for both generative and discriminative tasks, complemented by two mitigation baselines. Extensive evaluation on 12 mainstream MLLMs reveals substantial hallucinations under perturbed inputs and biases toward certain perturbations, notably image concatenation, image cropping, and prompt misleading. The work provides open data, tools, and metrics, underscoring the need for robust perturbed-input robustness and offering initial mitigation strategies that show partial success in reducing hallucinations.

Abstract

Multi-modal Large Language Models (MLLMs) have demonstrated remarkable performance on various visual-language understanding and generation tasks. However, MLLMs occasionally generate content inconsistent with the given images, which is known as "hallucination". Prior works primarily center on evaluating hallucination using standard, unperturbed benchmarks, which overlook the prevalent occurrence of perturbed inputs in real-world scenarios-such as image cropping or blurring-that are critical for a comprehensive assessment of MLLMs' hallucination. In this paper, to bridge this gap, we propose Hallu-PI, the first benchmark designed to evaluate Hallucination in MLLMs within Perturbed Inputs. Specifically, Hallu-PI consists of seven perturbed scenarios, containing 1,260 perturbed images from 11 object types. Each image is accompanied by detailed annotations, which include fine-grained hallucination types, such as existence, attribute, and relation. We equip these annotations with a rich set of questions, making Hallu-PI suitable for both discriminative and generative tasks. Extensive experiments on 12 mainstream MLLMs, such as GPT-4V and Gemini-Pro Vision, demonstrate that these models exhibit significant hallucinations on Hallu-PI, which is not observed in unperturbed scenarios. Furthermore, our research reveals a severe bias in MLLMs' ability to handle different types of hallucinations. We also design two baselines specifically for perturbed scenarios, namely Perturbed-Reminder and Perturbed-ICL. We hope that our study will bring researchers' attention to the limitations of MLLMs when dealing with perturbed inputs, and spur further investigations to address this issue. Our code and datasets are publicly available at https://github.com/NJUNLP/Hallu-PI.

Figures (11)

• Figure 1: Some examples of hallucinations in MLLMs with perturbed inputs (such as image concatenation, image cropping, and prompt misleading). Text highlighted in green and red represents correct and hallucinatory content, respectively.
• Figure 2: The hallucinatory performance of various MLLMs before (blue bars) and after (orange bars) input perturbation. Inconsistent performance trends show that relying solely on unperturbed benchmarks is insufficient for a complete and precise evaluation of hallucinations in MLLMs.
• Figure 3: (a) Overview of Hallu-PI pipeline for image annotation and perturbation. (b) An illustration of evaluation pipeline of Hallu-PI, including both generative and discriminative tasks.
• Figure 4: The data distribution of Hallu-PI.
• Figure 5: The performance variation (before and after image concatenation) in five annotated attributes.