Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction

Ruicheng Yuan, Zhenxuan Zhang, Anbang Wang, Liwei Hu, Xiangqian Hua, Yaya Peng, Jiawei Luo, Guang Yang

Abstract

Pathology reports are structured, multi-granular documents encoding diagnostic conclusions, histological grades, and ancillary test results across one or more anatomical sites; yet existing pathology vision-language models (VLMs) reduce this output to a flat label or free-form text. We present HiPath, a lightweight VLM framework built on frozen UNI2 and Qwen3 backbones that treats structured report prediction as its primary training objective. Three trainable modules totalling 15M parameters address complementary aspects of the problem: a Hierarchical Patch Aggregator (HiPA) for multi-image visual encoding, Hierarchical Contrastive Learning (HiCL) for cross-modal alignment via optimal transport, and Slot-based Masked Diagnosis Prediction (Slot-MDP) for structured diagnosis generation. Trained on 749K real-world Chinese pathology cases from three hospitals, HiPath achieves 68.9% strict and 74.7% clinically acceptable accuracy with a 97.3% safety rate, outperforming all baselines under the same frozen backbone. Cross-hospital evaluation confirms generalisation with only a 3.4pp drop in strict accuracy while maintaining 97.1% safety.

HiPath: Hierarchical Vision-Language Alignment for Structured Pathology Report Prediction

Abstract

Pathology reports are structured, multi-granular documents encoding diagnostic conclusions, histological grades, and ancillary test results across one or more anatomical sites; yet existing pathology vision-language models (VLMs) reduce this output to a flat label or free-form text. We present HiPath, a lightweight VLM framework built on frozen UNI2 and Qwen3 backbones that treats structured report prediction as its primary training objective. Three trainable modules totalling 15M parameters address complementary aspects of the problem: a Hierarchical Patch Aggregator (HiPA) for multi-image visual encoding, Hierarchical Contrastive Learning (HiCL) for cross-modal alignment via optimal transport, and Slot-based Masked Diagnosis Prediction (Slot-MDP) for structured diagnosis generation. Trained on 749K real-world Chinese pathology cases from three hospitals, HiPath achieves 68.9% strict and 74.7% clinically acceptable accuracy with a 97.3% safety rate, outperforming all baselines under the same frozen backbone. Cross-hospital evaluation confirms generalisation with only a 3.4pp drop in strict accuracy while maintaining 97.1% safety.
Paper Structure (7 sections, 7 equations, 1 figure, 3 tables)

This paper contains 7 sections, 7 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Hierarchical Patch Aggregator (HiPA)
  4. Hierarchical Contrastive Learning (HiCL)
  5. Slot-based Masked Diagnosis Prediction (Slot-MDP)
  6. Results and Discussion
  7. Conclusion

Figures (1)

  • Figure 1: Overview of HiPath. Frozen UNI2 patch features and Qwen3 text embeddings are processed by three trainable modules: HiPA aggregates patches hierarchically into case representations; HiCL aligns modalities at case and segment level via optimal transport; Slot-MDP fills typed diagnostic slots from visual features alone. Free-text reports are used only during training.