Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Project Hermes: A Model-Agnostic Validation Layer for Wearable Health Prediction Systems

Richik Chakraborty

TL;DR

Project Hermes is introduced, a model-agnostic validation layer that treats signal confirmation as a sequential decision problem, and establishes validation as a first-class computational problem distinct from prediction, with implications for trustworthy deployment of consumer health AI systems.

Abstract

The deployment of wearable-based health prediction systems has accelerated rapidly, yet these systems face a fundamental challenge: they generate alerts under substantial uncertainty without principled mechanisms for user-specific validation. While large language models (LLMs) have been increasingly applied to healthcare tasks, existing work focuses predominantly on diagnosis generation and risk prediction rather than post-prediction validation of detected signals. We introduce Project Hermes, a model-agnostic validation layer that treats signal confirmation as a sequential decision problem. Hermes operates downstream of arbitrary upstream predictors, using LLM-generated contextual queries to elicit targeted user feedback and performing Bayesian confidence updates to distinguish true positives from false alarms. In a 60-day longitudinal case study of migraine prediction, Hermes achieved a 34% reduction in false positive rate (from 61.7% to 12.5%) while maintaining 89% sensitivity, with mean lead time of 4.2 hours before symptom onset. Critically, Hermes does not perform diagnosis or make novel predictions; it validates whether signals detected by upstream models are clinically meaningful for specific individuals at specific times. This work establishes validation as a first-class computational problem distinct from prediction, with implications for trustworthy deployment of consumer health AI systems.

Project Hermes: A Model-Agnostic Validation Layer for Wearable Health Prediction Systems

TL;DR

Project Hermes is introduced, a model-agnostic validation layer that treats signal confirmation as a sequential decision problem, and establishes validation as a first-class computational problem distinct from prediction, with implications for trustworthy deployment of consumer health AI systems.

Abstract

The deployment of wearable-based health prediction systems has accelerated rapidly, yet these systems face a fundamental challenge: they generate alerts under substantial uncertainty without principled mechanisms for user-specific validation. While large language models (LLMs) have been increasingly applied to healthcare tasks, existing work focuses predominantly on diagnosis generation and risk prediction rather than post-prediction validation of detected signals. We introduce Project Hermes, a model-agnostic validation layer that treats signal confirmation as a sequential decision problem. Hermes operates downstream of arbitrary upstream predictors, using LLM-generated contextual queries to elicit targeted user feedback and performing Bayesian confidence updates to distinguish true positives from false alarms. In a 60-day longitudinal case study of migraine prediction, Hermes achieved a 34% reduction in false positive rate (from 61.7% to 12.5%) while maintaining 89% sensitivity, with mean lead time of 4.2 hours before symptom onset. Critically, Hermes does not perform diagnosis or make novel predictions; it validates whether signals detected by upstream models are clinically meaningful for specific individuals at specific times. This work establishes validation as a first-class computational problem distinct from prediction, with implications for trustworthy deployment of consumer health AI systems.
Paper Structure (85 sections, 42 equations, 3 figures, 4 tables)

This paper contains 85 sections, 42 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction: The Validation Gap in Wearable Health AI
  2. Related Work
  3. Wearable-Based Health Prediction
  4. Large Language Models in Healthcare
  5. Active Learning and Human-in-the-Loop Systems
  6. Positioning Hermes
  7. Conceptual Framing: Validation as a First-Class Problem
  8. Prediction versus Validation: A Formal Distinction
  9. Why Model-Agnosticism Matters
  10. Scope and Limitations by Design
  11. System Architecture: Project Hermes
  12. Upstream Predictor (External)
  13. Signal Triggering Layer
  14. Illness Codex
  15. LLM Question Generation Module
  16. ...and 70 more sections

Figures (3)

  • Figure 1: Hermes System Architecture. The system processes signals from an upstream predictor through six components: signal triggering evaluates whether validation should initiate, the illness codex provides structured medical knowledge, LLM question generation creates natural language queries, user responses provide evidence, Bayesian updating computes posterior confidence, and decision logic determines whether to alert, suppress, or ask additional questions.
  • Figure 2: Confidence trajectories for 47 validation instances over 60 days. Green lines represent true positives (n=16), blue lines show true negatives (n=27), red dashed lines indicate false negatives (n=2), and gray dash-dot lines show false positives (n=2). Horizontal lines mark alert threshold ($\tau = 0.70$) and suppression threshold ($\tau = 0.20$). Most trajectories resolve within 3 hours of signal detection.
  • Figure 3: Information gain per question across all validation instances. Box plots show distribution of information gain for question positions 1-5. First questions provide median 0.52 bits, declining to 0.15 bits by fifth question. Diminishing returns justify early termination when confidence crosses decision thresholds.