HERMES: A Unified Open-Source Framework for Realtime Multimodal Physiological Sensing, Edge AI, and Intervention in Closed-Loop Smart Healthcare Applications

TL;DR

The paper tackles the challenge of real-time, multimodal physiological sensing for real-world healthcare by proposing a holistic methodology and an open-source edge AI framework, HERMES. It introduces a continuous realtime multimodal edge AI approach with PUSH and PULL inference strategies and a continuous synchronization mechanism across distributed hosts, enabling reliable cross-modal fusion. The key contributions include a generalizable system methodology, a high-throughput streaming architecture with a flexible data alignment pipeline, and an open-source PyTorch-enabled framework validated on a distributed prosthesis-use case with 18 modalities and four hosts. The work demonstrates practical impact by providing concrete hardware measurements, latency budgets, and missingness analyses, guiding downstream AI model design toward robust, real-time interventions in clinical contexts. While not medically certified, it offers a concrete, reproducible path for researchers to prototype and evaluate closed-loop intelligent healthcare systems, with clear future directions for fault tolerance and broader sensor support.

Abstract

Intelligent assistive technologies are increasingly recognized as critical daily-use enablers for people with disabilities and age-related functional decline. Longitudinal studies, curation of quality datasets, live monitoring in activities of daily living, and intelligent intervention devices, share the largely unsolved need in reliable high-throughput multimodal sensing and processing. Streaming large heterogeneous data from distributed sensors, historically closed-source environments, and limited prior works on realtime closed-loop AI methodologies, inhibit such applications. To accelerate the emergence of clinical deployments, we deliver HERMES - an open-source high-performance Python framework for continuous multimodal sensing and AI processing at the edge. It enables synchronized data collection, and realtime streaming inference with user PyTorch models, on commodity computing devices. HERMES is applicable to fixed-lab and free-living environments, of distributed commercial and custom sensors. It is the first work to offer a holistic methodology that bridges cross-disciplinary gaps in real-world implementation strategies, and guides downstream AI model development. Its application on the closed-loop intelligent prosthesis use case illustrates the process of suitable AI model development from the generated constraints and trade-offs. Validation on the use case, with 4 synchronized hosts cooperatively capturing 18 wearable and off-body modalities, demonstrates performance and relevance of HERMES to the trajectory of the intelligent healthcare domain.

Figures (8)

• Figure 1: Generalization of multiple modalities (a): same data stream from multiple identical sensors (a.1), different sample rate data streams of the same sensor (a.2), separate same sample rate stream per bundle of measurements of the same sensor (a.3). Definition of realtime operation and latency in pipelined data flow (b). Two proposed extremes of continuous inference strategies: asynchronous event-driven (push) on any new modality update, and buffered alignment (pull) for the latest samples across modalities (c). Synchronization of distributed networked devices and connected sensors (d).
• Figure 2: Relationship between the AI receptive field and the overlap of consecutive windows. (Adapted from yudayev-rtstgcn-mlsp-2024.)
• Figure 3: Abstraction model that makes access of logic blocks to data of interest transparent across the network and devices, regardless of data locality (a). Decoupled system components at each host (b) - publisher (purple), consumer (green), pipeline (orange), fusion blocks responsible for data alignment and embedding strategies, storage component captures all sensed and processed data. Flow of message exchange based on topic data descriptors through a Broker (c).
• Figure 4: Efficient doubly-linked list FIFO datastructure for high-throughput low-latency streaming data. The Producer thread writes to it from the head and the Storage thread reads from it from the tail, simultaneously (a). Overview of the video data flow through the system (b). Memory pressure sawtooth profile - system keeps up with the streamed data if $\left|m_2\right| > \left|m_1\right|$: storage component sleeps during positive slope, flushes accumulated data during negative slope (c), adapted from ebner-memory-stack-2014. Visualization of the CPU usage in the system implementation leveraging multiprocessing, multithreading, and asynchronous IO coroutines (d).
• Figure 5: Flow path of a sensor sample through the system, stamped with the host's time on arrival (a). Inter-alignment of asynchronously received correlated measurements of distinct IMUs, experiencing congestion in wireless communication (b). Desynchronization between a 60 Hz sensor with an actual sample rate of 60.1 Hz, over a minute window, and a sensor with a perfect 60.0 Hz onboard clock: the former produces 6 samples more each minute (100 milliseconds worth), making direct one-to-one pairing of samples increasingly incorrect overtime, unless data is timed w.r.t. host's time (c).