A Dataset and Benchmark for Consumer Healthcare Question Summarization

TL;DR

The paper introduces CHQ-Summ, a dataset of 1507 domain-expert annotated consumer health question–summary pairs drawn from the Yahoo Answers L6 corpus, enriched with question_focus and question_type annotations to aid healthcare QA. It describes a MeSH-based mapping step that assigns MeSH headings to question focuses, enabling richer indexing and analysis of the dataset. The authors benchmark both fine-tuned encoder-decoder models (BART, PEGASUS, ProphetNet, T5-base) and instruction-tuned LLMs (e.g., Qwen2-7B-Instruct, Mistral-7B-Instruct, Llama-3, Gemma-7B-IT, DeepSeek-7B-Chat) under zero-/few-shot settings across four prompting strategies (Standard, Element-Aware, Hierarchical, Chain-of-Density), with both reference-based metrics (ROUGE-LSum, METEOR, BERTScore) and reference-free measures (Semantic Coherence, Entailment). Key results show ProphetNet-Large-Uncased achieving the strongest ROUGE-LSum and METEOR among fine-tuned models, BART-Large the highest BERTScore, and T5-Base the best Semantic Coherence, while PEGASUS-Large yields the top Entailment score; among instruction-tuned LLMs, Hierarchical prompting with smaller models often yields the best lexical overlap, whereas Element-Aware prompting excels in factual faithfulness. LiveQA-based retrieval experiments demonstrate that summarized questions (both reference and model-generated) improve answer retrieval performance compared to original questions, and human evaluation favors certain lightweight LLMs for factual correctness and fluency. The work provides data and code releases to promote supervised and zero-/few-shot evaluation of healthcare QA systems and underscores the value of MeSH-informed analysis for consumer health NLP.

Abstract

The quest for seeking health information has swamped the web with consumers health-related questions. Generally, consumers use overly descriptive and peripheral information to express their medical condition or other healthcare needs, contributing to the challenges of natural language understanding. One way to address this challenge is to summarize the questions and distill the key information of the original question. Recently, large-scale datasets have significantly propelled the development of several summarization tasks, such as multi-document summarization and dialogue summarization. However, a lack of a domain-expert annotated dataset for the consumer healthcare questions summarization task inhibits the development of an efficient summarization system. To address this issue, we introduce a new dataset, CHQ-Sum,m that contains 1507 domain-expert annotated consumer health questions and corresponding summaries. The dataset is derived from the community question answering forum and therefore provides a valuable resource for understanding consumer health-related posts on social media. We benchmark the dataset on multiple state-of-the-art summarization models to show the effectiveness of the dataset

Figures (7)

• Figure 1: Sample consumer healthcare question and reference summary with the annotated question focus and question type.
• Figure 2: The schematic workflow of the CHQ-Summ dataset creation.
• Figure 2: Dataset statistics for consumer health question and its corresponding summary in train, test and validation set.
• Figure 3: Distribution of words in original questions (a) and summarized questions (b) in CHQ-Summ dataset.
• Figure 4: Coverage of the MeSH headings (left) and question focus terms (right) on the CHQ-Summ dataset.