Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering

Ao Li, Rui Liu, Mingjie Li, Sheng Liu, Lei Wang, Xiaodan Liang, Lina Yao, Xiaojun Chang, Lei Xing

TL;DR

This work proposes a training-free, inference-time control framework termed Semantically Decoupled Latent Steering (SDLS), which constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition followed by QR-based orthogonalization.

Abstract

Automated radiology report generation using vision-language models (VLMs) is limited by the risk of prior-comparison hallucination, where the model generates historical findings unsupported by the current study. We address this challenge with a training-free, inference-time control framework termed Semantically Decoupled Latent Steering (SDLS). Unlike generic activation steering, which often suffers from semantic entanglement, our approach constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition followed by $QR$-based orthogonalization. This orthogonalization step is critical. It leverages geometric constraints to filter out the clinical semantics often entangled in standard principal component analysis (PCA) directions, ensuring that the steering vector targets only the ``historical comparison" axis. We validate our method on the BiomedGPT foundation model, demonstrating that it overcomes the trade-off between hallucination suppression and clinical accuracy. Extensive experiments on MIMIC-CXR, and zero-shot transfer evaluation on CheXpert Plus and IU-Xray, demonstrate the robustness of our approach. Quantitative evaluations on MIMIC-CXR show that our approach significantly reduces the probability of historical hallucinations (FilBERT score decreases from 0.2373 to 0.1889) and improves clinical label fidelity (CheXpert macro-F1 increases from 0.2242 to 0.3208). Supplementary evaluations confirm that the structural integrity of the clinical narrative is maintained.

Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering

TL;DR

This work proposes a training-free, inference-time control framework termed Semantically Decoupled Latent Steering (SDLS), which constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition followed by QR-based orthogonalization.

Abstract

Automated radiology report generation using vision-language models (VLMs) is limited by the risk of prior-comparison hallucination, where the model generates historical findings unsupported by the current study. We address this challenge with a training-free, inference-time control framework termed Semantically Decoupled Latent Steering (SDLS). Unlike generic activation steering, which often suffers from semantic entanglement, our approach constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition followed by -based orthogonalization. This orthogonalization step is critical. It leverages geometric constraints to filter out the clinical semantics often entangled in standard principal component analysis (PCA) directions, ensuring that the steering vector targets only the ``historical comparison" axis. We validate our method on the BiomedGPT foundation model, demonstrating that it overcomes the trade-off between hallucination suppression and clinical accuracy. Extensive experiments on MIMIC-CXR, and zero-shot transfer evaluation on CheXpert Plus and IU-Xray, demonstrate the robustness of our approach. Quantitative evaluations on MIMIC-CXR show that our approach significantly reduces the probability of historical hallucinations (FilBERT score decreases from 0.2373 to 0.1889) and improves clinical label fidelity (CheXpert macro-F1 increases from 0.2242 to 0.3208). Supplementary evaluations confirm that the structural integrity of the clinical narrative is maintained.
Paper Structure (64 sections, 10 equations, 5 figures, 7 tables, 1 algorithm)

This paper contains 64 sections, 10 equations, 5 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Radiology Report Generation: Architecture and Evaluation
  4. The Challenge of Temporal Hallucination in Medical AI
  5. Mitigation Strategies: Data-Centric vs. Latent Steering
  6. Data-Centric Interventions (Retraining)
  7. Inference-Time Representation Control
  8. Mechanistic Underpinnings: The Modality Gap
  9. Method
  10. Task and Scope
  11. SDIV Formulation via $QR$-based Semantic Decomposition
  12. Notation and Representation Basis
  13. Baseline ICV Formulations for Ablation
  14. Global ICV (Broad Distributional Bias)
  15. Curated-Specific-50 ICV (Surgical Removal)
  16. ...and 49 more sections

Figures (5)

  • Figure 1: Geometric Logic of SDLS. The latent space consists of two conflicting components: the Visual Manifold, which encodes grounded image features representing the clinical ground truth (e.g., current evidence of 'pleural effusion'), and the Bias Subspace, which captures statistical language priors (e.g., the tendency to generate 'no interval change' regardless of visual evidence). Standard steering produces an Entangled Naive Vector (dashed) that inadvertently shifts representations on the manifold, creating a Semantic Error. Our Semantically Decoupled Intervention Vector(SDIV) utilizes $QR$ decomposition to isolate the strictly Orthogonal Component, projecting representations vertically to remove bias without spatial distortion.
  • Figure 2: Schematic overview of the Semantically Decoupled Latent Steering (SDLS) framework. The framework comprises an offline preparation phase (a-b) and an online deployment phase (c). (a) Contrastive Context Mining: Paired reports (hallucinated vs. clean) are encoded to isolate the raw bias representation. (b) Orthogonal Latent Decomposition: A $QR$-based factorization extracts the pure hallucination direction (SDIV, purple arrow) orthogonal to image semantics. (c) Inference-Time Steering: During test time, the SDIV is injected into decoder layers to suppress prior-comparison tokens without model retraining.
  • Figure 3: Qualitative Visualization of Visual Grounding. We visualize the cross-attention maps during the generation of specific target tokens. (a, b) Factual terms: For grounded terms like "endotracheal" or "cardiopulmonary", the model's attention is strongly focused on the relevant anatomical regions (trachea, heart), indicating robust visual grounding. (c, d) Hallucinations: For historical comparison terms like "unchanged" or "stable", the attention map is diffuse and failed to focus on meaningful visual features, confirming that these words are driven by language priors rather than visual evidence.
  • Figure 4: Dose-response curve showing FilBERT Score as a function of intervention strength ($\lambda$) for SDIV on BiomedGPT. The curve demonstrates a clear optimal intervention zone, highlighting the controllability of the method.
  • Figure 5: Robustness and specificity checks. Standard negative controls (random, shuffled) show no suppression effect. Note that the style-orthogonalized ICV (orange dashed) underperforms the original baseline.