Tone Matters: The Impact of Linguistic Tone on Hallucination in VLMs

TL;DR

This work investigates how linguistic tone affects hallucination in Vision–Language Models when visual evidence is missing. It introduces Ghost-100, a diagnostic dataset, and a five-level Prompt Intensity Framework to separate semantic toxicity from structural coercion, evaluated across three open-weight VLMs in a zero-shot setup. The study reveals non-monotonic hallucination behavior: stronger prompts can trigger refusals or uncertainty rather than sustained fabrication, with effects varying by model and prompt form. The findings highlight the need for alignment approaches that balance instruction compliance with explicit uncertainty or refusal when visual grounding is absent, and they provide a reproducible benchmark for safety evaluation in multimodal systems.

Abstract

Vision-Language Models (VLMs) are increasingly used in safety-critical applications that require reliable visual grounding. However, these models often hallucinate details that are not present in the image to satisfy user prompts. While recent datasets and benchmarks have been introduced to evaluate systematic hallucinations in VLMs, many hallucination behaviors remain insufficiently characterized. In particular, prior work primarily focuses on object presence or absence, leaving it unclear how prompt phrasing and structural constraints can systematically induce hallucinations. In this paper, we investigate how different forms of prompt pressure influence hallucination behavior. We introduce Ghost-100, a procedurally generated dataset of synthetic scenes in which key visual details are deliberately removed, enabling controlled analysis of absence-based hallucinations. Using a structured 5-Level Prompt Intensity Framework, we vary prompts from neutral queries to toxic demands and rigid formatting constraints. We evaluate three representative open-weight VLMs: MiniCPM-V 2.6-8B, Qwen2-VL-7B, and Qwen3-VL-8B. Across all three models, hallucination rates do not increase monotonically with prompt intensity. All models exhibit reductions at higher intensity levels at different thresholds, though not all show sustained reduction under maximum coercion. These results suggest that current safety alignment is more effective at detecting semantic hostility than structural coercion, revealing model-specific limitations in handling compliance pressure. Our dataset is available at: https://github.com/bli1/tone-matters

Figures (3)

• Figure 1: Non-Monotonic Hallucination Across Prompt Intensity Levels: An Illustrative Example.
• Figure 2: Hallucination Rates under Increasing Prompt Intensity
• Figure 3: Hallucination Scores under Increasing Tones Intensity