Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answering

Awais Naeem, Tianhao Li, Huang-Ru Liao, Jiawei Xu, Aby M. Mathew, Zehao Zhu, Zhen Tan, Ajay Kumar Jaiswal, Raffi A. Salibian, Ziniu Hu, Tianlong Chen, Ying Ding

TL;DR

Path-RAG tackles open-ended pathology visual question answering by injecting domain knowledge through HistoCartography-based patch retrieval. It combines nucleus-focused patch extraction, patch-level descriptions from LLaVA-Med, and GPT-4-based textual reasoning to produce robust answers without heavy LLM fine-tuning. The approach yields state-of-the-art recall on PathVQA-Open (up to 47.4%), and large gains on ARCH-Open long-form QA for both PubMed and Books datasets, especially on H&E images. This domain-guided retrieval framework demonstrates the value of structured tissue-region knowledge in multi-modal pathology reasoning and offers reusable datasets and code for reproducibility.

Abstract

Accurate diagnosis and prognosis assisted by pathology images are essential for cancer treatment selection and planning. Despite the recent trend of adopting deep-learning approaches for analyzing complex pathology images, they fall short as they often overlook the domain-expert understanding of tissue structure and cell composition. In this work, we focus on a challenging Open-ended Pathology VQA (PathVQA-Open) task and propose a novel framework named Path-RAG, which leverages HistoCartography to retrieve relevant domain knowledge from pathology images and significantly improves performance on PathVQA-Open. Admitting the complexity of pathology image analysis, Path-RAG adopts a human-centered AI approach by retrieving domain knowledge using HistoCartography to select the relevant patches from pathology images. Our experiments suggest that domain guidance can significantly boost the accuracy of LLaVA-Med from 38% to 47%, with a notable gain of 28% for H&E-stained pathology images in the PathVQA-Open dataset. For longer-form question and answer pairs, our model consistently achieves significant improvements of 32.5% in ARCH-Open PubMed and 30.6% in ARCH-Open Books on H\&E images. Our code and dataset is available here (https://github.com/embedded-robotics/path-rag).

Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answering

TL;DR

Path-RAG tackles open-ended pathology visual question answering by injecting domain knowledge through HistoCartography-based patch retrieval. It combines nucleus-focused patch extraction, patch-level descriptions from LLaVA-Med, and GPT-4-based textual reasoning to produce robust answers without heavy LLM fine-tuning. The approach yields state-of-the-art recall on PathVQA-Open (up to 47.4%), and large gains on ARCH-Open long-form QA for both PubMed and Books datasets, especially on H&E images. This domain-guided retrieval framework demonstrates the value of structured tissue-region knowledge in multi-modal pathology reasoning and offers reusable datasets and code for reproducibility.

Abstract

Accurate diagnosis and prognosis assisted by pathology images are essential for cancer treatment selection and planning. Despite the recent trend of adopting deep-learning approaches for analyzing complex pathology images, they fall short as they often overlook the domain-expert understanding of tissue structure and cell composition. In this work, we focus on a challenging Open-ended Pathology VQA (PathVQA-Open) task and propose a novel framework named Path-RAG, which leverages HistoCartography to retrieve relevant domain knowledge from pathology images and significantly improves performance on PathVQA-Open. Admitting the complexity of pathology image analysis, Path-RAG adopts a human-centered AI approach by retrieving domain knowledge using HistoCartography to select the relevant patches from pathology images. Our experiments suggest that domain guidance can significantly boost the accuracy of LLaVA-Med from 38% to 47%, with a notable gain of 28% for H&E-stained pathology images in the PathVQA-Open dataset. For longer-form question and answer pairs, our model consistently achieves significant improvements of 32.5% in ARCH-Open PubMed and 30.6% in ARCH-Open Books on H\&E images. Our code and dataset is available here (https://github.com/embedded-robotics/path-rag).

Paper Structure

This paper contains 19 sections, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. H&E stained Pathology Image Identification
  4. Pathology Image Decomposition
  5. Patch Captioning and Candidate Answer Extraction
  6. Reasoning over Per-Patch Extracted Knowledge
  7. Experimental Setup
  8. PathVQA Dataset.
  9. ARCH-Open Dataset.
  10. Implementation Settings
  11. Results and Ablation Study
  12. Comparison with SOTA:
  13. Effectiveness of Retrieving Patches with HistoCartography:
  14. Effect of textual reasoning:
  15. Effect of different number of patches
  16. ...and 4 more sections

Figures (7)

  • Figure 1: The overview of our Path-RAG framework.
  • Figure 2: Word Count distribution of open-ended answers in PathVQA
  • Figure 3: Word Count distribution in ARCH-Open Questions - PubMed
  • Figure 4: Word Count distribution in ARCH-Open Answers - PubMed
  • Figure 5: Word Count distribution in ARCH-Open Questions - Books
  • ...and 2 more figures