Hearing Health in Home Healthcare: Leveraging LLMs for Illness Scoring and ALMs for Vocal Biomarker Extraction

TL;DR

The paper addresses the challenge of automatic, scalable health assessment from real-world in-home voice data by developing a dual approach: an LLM-based holistic illness score that fuses SOAP notes with vital signs into a compact metric $S \in \{1,\dots,5\}$, and an ALM-based pipeline that extracts interpretable vocal biomarkers from short speaker segments. Using 160 home-care visits, the study shows SOAP notes carry more health signal than vital signs, with GPT-4.1 achieving strong alignment to clinical outcomes; it also provides the first evidence that ALMs can identify health-related acoustic patterns and describe them in plain language, with descriptors correlating to illness scores. The work demonstrates a practical, interpretable multimodal framework for monitoring patients in home care, potentially aiding clinicians with early detection and continuous tracking of health status through voice. Together, these findings motivate further integration of LLMs and ALMs in multimodal health surveillance and call for more robust validation and extension to broader health conditions.

Abstract

The growing demand for home healthcare calls for tools that can support care delivery. In this study, we explore automatic health assessment from voice using real-world home care visit data, leveraging the diverse patient information it contains. First, we utilize Large Language Models (LLMs) to integrate Subjective, Objective, Assessment, and Plan (SOAP) notes derived from unstructured audio transcripts and structured vital signs into a holistic illness score that reflects a patient's overall health. This compact representation facilitates cross-visit health status comparisons and downstream analysis. Next, we design a multi-stage preprocessing pipeline to extract short speech segments from target speakers in home care recordings for acoustic analysis. We then employ an Audio Language Model (ALM) to produce plain-language descriptions of vocal biomarkers and examine their association with individuals' health status. Our experimental results benchmark both commercial and open-source LLMs in estimating illness scores, demonstrating their alignment with actual clinical outcomes, and revealing that SOAP notes are substantially more informative than vital signs. Building on the illness scores, we provide the first evidence that ALMs can identify health-related acoustic patterns from home care recordings and present them in a human-readable form. Together, these findings highlight the potential of LLMs and ALMs to harness heterogeneous in-home visit data for better patient monitoring and care.

Figures (14)

• Figure 1: Overview of our analysis framework. Home care visit audio recordings are processed through two pipelines. Top: An LLM summarizes ASR transcripts into SOAP notes and combines them with vital signs to produce an illness score. Bottom: Preprocessed patient speech segments are analyzed by an ALM to generate acoustic descriptors, which are correlated with the illness score.
• Figure 2: KDE plots of generated illness score distributions of No ED/HOSP and ED/HOSP groups. Each row shows results for a specific LLM, while columns represent different inputs. Across most settings, ED/HOSP groups show higher average scores than No ED/HOSP groups, indicating alignment with clinical outcomes. SOAP-based inputs (SOAP only and SOAP+VITAL) show better group separation than VITAL only, with consistently higher scores for ED/HOSP patients.
• Figure 3: Wasserstein distances between illness score distributions of No ED/HOSP and ED/HOSP groups across four models. A higher Wasserstein distance indicates better separation between groups. SOAP notes provide substantially more discriminative information than vital signs alone. GPT-4.1 achieves the highest average performance and benefits from incorporating vital signs, while Qwen3-8B and Llama3.1-8B show degraded performance when vitals are added.
• Figure 4: GPT-4o-audio performance on a baseline speaker classification tasks. Left: Confusion matrix showing GPT-4o-audio achieves 0.8625 accuracy on gender detection. Right: Age prediction boxplot showing predicted age categories that align well with true ages.
• Figure 5: Wordcloud of GPT-4o-audio generated phrase frequencies for clinician and patient voices.