Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Demo: Guide-RAG: Evidence-Driven Corpus Curation for Retrieval-Augmented Generation in Long COVID

Philip DiGiacomo, Haoyang Wang, Jinrui Fang, Yan Leng, W Michael Brode, Ying Ding

TL;DR

This work addresses the challenge of producing trustworthy clinical QA for emerging diseases by evaluating six Retrieval-Augmented Generation corpus configurations for Long COVID. It introduces Guide-RAG, an evidence-grounded chatbot framework, and LongCOVID-CQ, a specialized evaluation dataset generated by clinicians. The study finds that grounding responses in a synthesized LC guideline plus high-quality systematic reviews (GS-4) yields the best overall fidelity, relevance, and coverage, outperforming large unfiltered literature bases. The results support a design principle that curated secondary reviews strike a balance between consensus guidance and primary literature, reducing information overload while maintaining clinical usefulness. The framework and evaluation approach offer a transferable methodology for building decision-support tools in other complex, evolving medical conditions.

Abstract

As AI chatbots gain adoption in clinical medicine, developing effective frameworks for complex, emerging diseases presents significant challenges. We developed and evaluated six Retrieval-Augmented Generation (RAG) corpus configurations for Long COVID (LC) clinical question answering, ranging from expert-curated sources to large-scale literature databases. Our evaluation employed an LLM-as-a-judge framework across faithfulness, relevance, and comprehensiveness metrics using LongCOVID-CQ, a novel dataset of expert-generated clinical questions. Our RAG corpus configuration combining clinical guidelines with high-quality systematic reviews consistently outperformed both narrow single-guideline approaches and large-scale literature databases. Our findings suggest that for emerging diseases, retrieval grounded in curated secondary reviews provides an optimal balance between narrow consensus documents and unfiltered primary literature, supporting clinical decision-making while avoiding information overload and oversimplified guidance. We propose Guide-RAG, a chatbot system and accompanying evaluation framework that integrates both curated expert knowledge and comprehensive literature databases to effectively answer LC clinical questions.

Demo: Guide-RAG: Evidence-Driven Corpus Curation for Retrieval-Augmented Generation in Long COVID

TL;DR

This work addresses the challenge of producing trustworthy clinical QA for emerging diseases by evaluating six Retrieval-Augmented Generation corpus configurations for Long COVID. It introduces Guide-RAG, an evidence-grounded chatbot framework, and LongCOVID-CQ, a specialized evaluation dataset generated by clinicians. The study finds that grounding responses in a synthesized LC guideline plus high-quality systematic reviews (GS-4) yields the best overall fidelity, relevance, and coverage, outperforming large unfiltered literature bases. The results support a design principle that curated secondary reviews strike a balance between consensus guidance and primary literature, reducing information overload while maintaining clinical usefulness. The framework and evaluation approach offer a transferable methodology for building decision-support tools in other complex, evolving medical conditions.

Abstract

As AI chatbots gain adoption in clinical medicine, developing effective frameworks for complex, emerging diseases presents significant challenges. We developed and evaluated six Retrieval-Augmented Generation (RAG) corpus configurations for Long COVID (LC) clinical question answering, ranging from expert-curated sources to large-scale literature databases. Our evaluation employed an LLM-as-a-judge framework across faithfulness, relevance, and comprehensiveness metrics using LongCOVID-CQ, a novel dataset of expert-generated clinical questions. Our RAG corpus configuration combining clinical guidelines with high-quality systematic reviews consistently outperformed both narrow single-guideline approaches and large-scale literature databases. Our findings suggest that for emerging diseases, retrieval grounded in curated secondary reviews provides an optimal balance between narrow consensus documents and unfiltered primary literature, supporting clinical decision-making while avoiding information overload and oversimplified guidance. We propose Guide-RAG, a chatbot system and accompanying evaluation framework that integrates both curated expert knowledge and comprehensive literature databases to effectively answer LC clinical questions.
Paper Structure (21 sections, 3 figures)

This paper contains 21 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Corpus construction
  5. Dataset: LongCOVID-CQ
  6. Evaluation framework
  7. Results
  8. Discussion and Conclusion
  9. Limitations
  10. Non-Pharmacologic Management
  11. Pharmacologic Management
  12. Referral and Specialist Consultation
  13. Holistic Management
  14. Non-Pharmacological Interventions
  15. Pharmacological Interventions
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Workflow of Guide-RAG answering LC questions. The full prompt used for the answer generation and evaluation is presented in Appendix A.
  • Figure 2: Win rates for (A) faithfulness, (B) relevance, (C) comprehensiveness, and (D) overall (each cell is the percentage of comparisons won by the row label over the column label)
  • Figure 3: Comparison of answers to an example question from the LongCOVID-CQ dataset using the GS-4, R-110, and PM corpuses. Each response's section headers are included in the figure for conciseness.