Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Health Information Retrieval with RAG by Prioritizing Topical Relevance and Factual Accuracy

Rishabh Uapadhyay, Marco Viviani

TL;DR

The paper addresses the risk of health misinformation in online information retrieval by proposing a Retrieval-Augmented Generation framework that grounds health document retrieval in scientific evidence. It introduces a three-stage pipeline: (1) topic-focused passage retrieval from PubMed Central using BM25 and BioBERT-based embeddings with NER-based discounts, (2) GenText generation from LLMs conditioned on the query and retrieved passages, and (3) a joint topicality and factual accuracy ranking (RSV) that guides final document selection. Factual accuracy is quantified through a combination of stance detection and semantic similarity to GenText, with explainability embedded via GenText and its citations. Experimental results on CLEF eHealth 2020 and TREC Health Misinformation 2020 show that RAG-driven variants outperform traditional IR baselines, highlighting the approach's potential to reduce health misinformation and provide transparent justifications for retrieved results.

Abstract

The exponential surge in online health information, coupled with its increasing use by non-experts, highlights the pressing need for advanced Health Information Retrieval models that consider not only topical relevance but also the factual accuracy of the retrieved information, given the potential risks associated with health misinformation. To this aim, this paper introduces a solution driven by Retrieval-Augmented Generation (RAG), which leverages the capabilities of generative Large Language Models (LLMs) to enhance the retrieval of health-related documents grounded in scientific evidence. In particular, we propose a three-stage model: in the first stage, the user's query is employed to retrieve topically relevant passages with associated references from a knowledge base constituted by scientific literature. In the second stage, these passages, alongside the initial query, are processed by LLMs to generate a contextually relevant rich text (GenText). In the last stage, the documents to be retrieved are evaluated and ranked both from the point of view of topical relevance and factual accuracy by means of their comparison with GenText, either through stance detection or semantic similarity. In addition to calculating factual accuracy, GenText can offer a layer of explainability for it, aiding users in understanding the reasoning behind the retrieval. Experimental evaluation of our model on benchmark datasets and against baseline models demonstrates its effectiveness in enhancing the retrieval of both topically relevant and factually accurate health information, thus presenting a significant step forward in the health misinformation mitigation problem.

Enhancing Health Information Retrieval with RAG by Prioritizing Topical Relevance and Factual Accuracy

TL;DR

The paper addresses the risk of health misinformation in online information retrieval by proposing a Retrieval-Augmented Generation framework that grounds health document retrieval in scientific evidence. It introduces a three-stage pipeline: (1) topic-focused passage retrieval from PubMed Central using BM25 and BioBERT-based embeddings with NER-based discounts, (2) GenText generation from LLMs conditioned on the query and retrieved passages, and (3) a joint topicality and factual accuracy ranking (RSV) that guides final document selection. Factual accuracy is quantified through a combination of stance detection and semantic similarity to GenText, with explainability embedded via GenText and its citations. Experimental results on CLEF eHealth 2020 and TREC Health Misinformation 2020 show that RAG-driven variants outperform traditional IR baselines, highlighting the approach's potential to reduce health misinformation and provide transparent justifications for retrieved results.

Abstract

The exponential surge in online health information, coupled with its increasing use by non-experts, highlights the pressing need for advanced Health Information Retrieval models that consider not only topical relevance but also the factual accuracy of the retrieved information, given the potential risks associated with health misinformation. To this aim, this paper introduces a solution driven by Retrieval-Augmented Generation (RAG), which leverages the capabilities of generative Large Language Models (LLMs) to enhance the retrieval of health-related documents grounded in scientific evidence. In particular, we propose a three-stage model: in the first stage, the user's query is employed to retrieve topically relevant passages with associated references from a knowledge base constituted by scientific literature. In the second stage, these passages, alongside the initial query, are processed by LLMs to generate a contextually relevant rich text (GenText). In the last stage, the documents to be retrieved are evaluated and ranked both from the point of view of topical relevance and factual accuracy by means of their comparison with GenText, either through stance detection or semantic similarity. In addition to calculating factual accuracy, GenText can offer a layer of explainability for it, aiding users in understanding the reasoning behind the retrieval. Experimental evaluation of our model on benchmark datasets and against baseline models demonstrates its effectiveness in enhancing the retrieval of both topically relevant and factually accurate health information, thus presenting a significant step forward in the health misinformation mitigation problem.

Paper Structure

This paper contains 13 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. A RAG-Driven Model for Health Information Retrieval
  4. User Query-Based Passage Retrieval from PMC
  5. GenText Generation through LLMs
  6. Calculating Topicality and Factual Accuracy, and Final Document Ranking
  7. Experimental Evaluation
  8. Datasets
  9. Implementation Details
  10. Baselines
  11. Experimental Results
  12. Enhancing Explainability with GenText
  13. Conclusion and Discussion

Figures (3)

  • Figure 1: The three stages underlying the proposed RAG-driven model for Health Information Retrieval.
  • Figure 3: Example of an instance of the prompt guiding LLMs.
  • Figure 4: Example of GenText used for explainability, addressing a query about the relationship between 5G technology and COVID-19 with references to scientific literature.