A Dataset and Resources for Identifying Patient Health Literacy Information from Clinical Notes

Abstract

Health literacy is a critical determinant of patient outcomes, yet current screening tools are not always feasible and differ considerably in the number of items, question format, and dimensions of health literacy they capture, making documentation in structured electronic health records difficult to achieve. Automated detection from unstructured clinical notes offers a promising alternative, as these notes often contain richer, more contextual health literacy information, but progress has been limited by the lack of annotated resources. We introduce HEALIX, the first publicly available annotated health literacy dataset derived from real clinical notes, curated through a combination of social worker note sampling, keyword-based filtering, and LLM-based active learning. HEALIX contains 589 notes across 9 note types, annotated with three health literacy labels: low, normal, and high. To demonstrate its utility, we benchmarked zero-shot and few-shot prompting strategies across four open source large language models (LLMs).

Figures (3)

• Figure 1: Workflow for dataset development. First, clinical notes were collected from the MIMIC-III database via three sampling strategies: random sampling of social worker notes, keyword-based filtering, and LLM-based active learning. Next, the sampled notes were independently annotated and reconciled to produce the HEALIX gold standard dataset. Finally, zero-shot and few-shot prompting strategies were applied to establish baseline model performance.
• Figure 2: Confusion matrices for SVM and LLaMA 70B (zero-/few-shot). SVM matrix uses a different color bar than the LLaMA matrices.
• Figure 3: Annotated clinical note excerpts illustrating examples of varying model classification difficulty. Highlighted text represents evidence of patient health literacy within the note. Any misspellings or abbreviations are reported as they appear in the original note.