HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoring

TL;DR

Privacy and latency barriers of cloud-based LLMs for mobile and wearable health monitoring motivate the exploration of on-device Small Language Models (SLMs). HealthSLM-Bench benchmarks nine SLMs on eight health-prediction tasks across three wearable datasets, under zero-shot, few-shot, and instruction-tuning paradigms, followed by on-device deployment to measure efficiency. Results show SLMs can match or exceed SOTA LLMs in several tasks while delivering substantial gains in memory and latency, though challenges persist in class imbalance and few-shot robustness. The work demonstrates the feasibility of private, real-time health monitoring on mobile devices and outlines future directions for robust prompt design, imbalance-aware training, and test-time adaptation.

Abstract

Mobile and wearable healthcare monitoring play a vital role in facilitating timely interventions, managing chronic health conditions, and ultimately improving individuals' quality of life. Previous studies on large language models (LLMs) have highlighted their impressive generalization abilities and effectiveness in healthcare prediction tasks. However, most LLM-based healthcare solutions are cloud-based, which raises significant privacy concerns and results in increased memory usage and latency. To address these challenges, there is growing interest in compact models, Small Language Models (SLMs), which are lightweight and designed to run locally and efficiently on mobile and wearable devices. Nevertheless, how well these models perform in healthcare prediction remains largely unexplored. We systematically evaluated SLMs on health prediction tasks using zero-shot, few-shot, and instruction fine-tuning approaches, and deployed the best performing fine-tuned SLMs on mobile devices to evaluate their real-world efficiency and predictive performance in practical healthcare scenarios. Our results show that SLMs can achieve performance comparable to LLMs while offering substantial gains in efficiency and privacy. However, challenges remain, particularly in handling class imbalance and few-shot scenarios. These findings highlight SLMs, though imperfect in their current form, as a promising solution for next-generation, privacy-preserving healthcare monitoring.

Figures (6)

• Figure 1: The label distribution of the four tasks in PMData
• Figure 2: The label distribution of the two tasks in GLOBEM
• Figure 3: The label distribution of the two tasks in AW_FB
• Figure 4: Distribution of predictions for the four tasks in PMData under FS setting
• Figure 5: Distribution of predictions for the four tasks in PMData