Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mind the Gap: Aligning Knowledge Bases with User Needs to Enhance Mental Health Retrieval

Amanda Chan, James Jiayu Liu, He Kai, Onno P. Kampman

TL;DR

The paper tackles the mismatch between user needs and mental health knowledge bases, which impairs retrieval quality in high-stakes contexts. It introduces a gap-informed corpus augmentation framework that overlays real user language data to identify Coverage Gap and Usefulness Gap, integrating them into a Hybrid Gap metric. The methodology yields directed synthetic content to fill identified gaps, evaluated across four RAG pipelines on mindline.sg with data from Let’s Talk forum; directed augmentation approaches near-exhaustive reference performance with far fewer documents, while non-directed expansions require orders of magnitude more. This work demonstrates a scalable, resource-efficient approach to building trusted mental health information repositories and can guide AI-enabled retrieval in other domain areas, emphasizing targeted content growth over indiscriminate expansion.

Abstract

Access to reliable mental health information is vital for early help-seeking, yet expanding knowledge bases is resource-intensive and often misaligned with user needs. This results in poor performance of retrieval systems when presented concerns are not covered or expressed in informal or contextualized language. We present an AI-based gap-informed framework for corpus augmentation that authentically identifies underrepresented topics (gaps) by overlaying naturalistic user data such as forum posts in order to prioritize expansions based on coverage and usefulness. In a case study, we compare Directed (gap-informed augmentations) with Non-Directed augmentation (random additions), evaluating the relevance and usefulness of retrieved information across four retrieval-augmented generation (RAG) pipelines. Directed augmentation achieved near-optimal performance with modest expansions--requiring only a 42% increase for Query Transformation, 74% for Reranking and Hierarchical, and 318% for Baseline--to reach ~95% of the performance of an exhaustive reference corpus. In contrast, Non-Directed augmentation required substantially larger and thus practically infeasible expansions to achieve comparable performance (232%, 318%, 403%, and 763%, respectively). These results show that strategically targeted corpus growth can reduce content creation demands while sustaining high retrieval and provision quality, offering a scalable approach for building trusted health information repositories and supporting generative AI applications in high-stakes domains.

Mind the Gap: Aligning Knowledge Bases with User Needs to Enhance Mental Health Retrieval

TL;DR

The paper tackles the mismatch between user needs and mental health knowledge bases, which impairs retrieval quality in high-stakes contexts. It introduces a gap-informed corpus augmentation framework that overlays real user language data to identify Coverage Gap and Usefulness Gap, integrating them into a Hybrid Gap metric. The methodology yields directed synthetic content to fill identified gaps, evaluated across four RAG pipelines on mindline.sg with data from Let’s Talk forum; directed augmentation approaches near-exhaustive reference performance with far fewer documents, while non-directed expansions require orders of magnitude more. This work demonstrates a scalable, resource-efficient approach to building trusted mental health information repositories and can guide AI-enabled retrieval in other domain areas, emphasizing targeted content growth over indiscriminate expansion.

Abstract

Access to reliable mental health information is vital for early help-seeking, yet expanding knowledge bases is resource-intensive and often misaligned with user needs. This results in poor performance of retrieval systems when presented concerns are not covered or expressed in informal or contextualized language. We present an AI-based gap-informed framework for corpus augmentation that authentically identifies underrepresented topics (gaps) by overlaying naturalistic user data such as forum posts in order to prioritize expansions based on coverage and usefulness. In a case study, we compare Directed (gap-informed augmentations) with Non-Directed augmentation (random additions), evaluating the relevance and usefulness of retrieved information across four retrieval-augmented generation (RAG) pipelines. Directed augmentation achieved near-optimal performance with modest expansions--requiring only a 42% increase for Query Transformation, 74% for Reranking and Hierarchical, and 318% for Baseline--to reach ~95% of the performance of an exhaustive reference corpus. In contrast, Non-Directed augmentation required substantially larger and thus practically infeasible expansions to achieve comparable performance (232%, 318%, 403%, and 763%, respectively). These results show that strategically targeted corpus growth can reduce content creation demands while sustaining high retrieval and provision quality, offering a scalable approach for building trusted health information repositories and supporting generative AI applications in high-stakes domains.

Paper Structure

This paper contains 24 sections, 2 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Data Collection and Preprocessing
  4. Topic Taxonomy and Annotation
  5. Content Gap Analysis
  6. Coverage Gap: TF-IDF–Inspired Scoring
  7. Usefulness Gap: LLM-as-a-Judge
  8. Hybrid Metric
  9. Synthetic Document Generation and Corpus Augmentation
  10. Construction of Directed and Non-Directed Corpora
  11. Automated Evaluation of Retrieval Quality
  12. Retrieval Pipelines
  13. Evaluation Method
  14. Results
  15. Discussion
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Overview of gap-informed corpus augmentation and evaluation framework. Forum posts are used to assess gaps in a mental health resource library, which is then augmented with additional articles and evaluated through multiple retrieval techniques.
  • Figure 2: Proportional distribution of Directed corpus across main topics.
  • Figure 3: LLM usefulness scores by corpus size, comparing Directed vs. Non-Directed retrieval.