Do More Details Always Introduce More Hallucinations in LVLM-based Image Captioning?

TL;DR

This work re-examines whether increasing detail in LVLM-based image captions inherently raises object hallucinations. It introduces Differentiated Beam Decoding (DBD), a training-free decoding strategy that generates diverse unit facts in parallel and selects a concise, non-redundant set for final captioning, guided by a differential score that promotes factual diversity. Accompanying this, the authors propose CLIP-Recall, CLIP-Precision, and CLIP-F1 to measure caption comprehensiveness and alignment with the image across partitions, addressing biases in prior metrics. Through extensive experiments on Visual Genome with LVLMs such as LLaVA-1.5, mPLUG-Owl2, and MiniGPT-4, the results show that richer descriptions can be produced with low hallucination, and that the CLIP-based metrics provide a more reliable evaluation of detail and fidelity. Overall, the work demonstrates that better-guided decoding can yield detailed, accurate captions, with practical implications for safety-critical multimodal tasks.

Abstract

Large Vision-Language Models (LVLMs) excel in integrating visual and linguistic contexts to produce detailed content, facilitating applications such as image captioning. However, using LVLMs to generate descriptions often faces the challenge of object hallucination (OH), where the output text misrepresents actual objects in the input image. While previous studies attribute the occurrence of OH to the inclusion of more details, our study finds technical flaws in existing metrics, leading to unreliable evaluations of models and conclusions about OH. This has sparked a debate on the question: Do more details always introduce more hallucinations in LVLM-based image captioning? In this paper, we address this debate by proposing a novel decoding strategy, Differentiated Beam Decoding (DBD), along with a reliable new set of evaluation metrics: CLIP-Precision, CLIP-Recall, and CLIP-F1. DBD decodes the wealth of information hidden in visual input into distinct language representations called unit facts in parallel. This decoding is achieved via a well-designed differential score that guides the parallel search and candidate screening. The selected unit facts are then aggregated to generate the final caption. Our proposed metrics evaluate the comprehensiveness and accuracy of image captions by comparing the embedding groups of ground-truth image regions and generated text partitions. Extensive experiments on the Visual Genome dataset validate the effectiveness of our approach, demonstrating that it produces detailed descriptions while maintaining low hallucination levels.

Figures (10)

• Figure 1: Comparison of captions. The top caption ignores much visual information. The middle one includes details but also introduces hallucinations. Our method (bottom) provides detailed and accurate captions.
• Figure 2: Object word position in the caption v.s. object size proportion to the image. As the caption progresses, LVLMs are prone to describe smaller objects.
• Figure 3: Misleading from pre-generated texts. With prior output engaged, LLaVA-1.5 predicts most likely five people in the scene. However, with the first sentence masked during the coding, the number decreases to four.
• Figure 4: Misjudgement from rule-based identification. The current framework misunderstands elaborate captions, where many object words appear not for existence but for decorative features or even for non-existence.
• Figure 5: Misjudgement from undetailed ground truths. The current framework identifies hallucination based on language ground truths, restricting the capability to evaluate captions in more detail than ground truths.