REMONI: An Autonomous System Integrating Wearables and Multimodal Large Language Models for Enhanced Remote Health Monitoring

TL;DR

The paper tackles clinician workload and gaps in professional-facing remote health tools by introducing REMONI, an autonomous system that fuses wearables, cameras, IoT, and multimodal language models. It integrates anomaly detection (fall and vital-sign monitoring) with a natural language processing engine to provide clinicians real-time status, activity, and mood information through a web interface, guided by a carefully engineered prompt system. Key contributions include a FallAllD-based HDL fall detector with high accuracy, a threshold-based vital-sign alert mechanism, a multimodal NLP pipeline for clinician queries, and an edge-cloud IoT deployment that demonstrates feasibility and scalability for real-world telehealth workflows. The results suggest REMONI can reduce healthcare workers’ workload and costs while maintaining timely alerts and informative interactions, with a prototype deployed and tested on practical hardware and cloud infrastructure.

Abstract

With the widespread adoption of wearable devices in our daily lives, the demand and appeal for remote patient monitoring have significantly increased. Most research in this field has concentrated on collecting sensor data, visualizing it, and analyzing it to detect anomalies in specific diseases such as diabetes, heart disease and depression. However, this domain has a notable gap in the aspect of human-machine interaction. This paper proposes REMONI, an autonomous REmote health MONItoring system that integrates multimodal large language models (MLLMs), the Internet of Things (IoT), and wearable devices. The system automatically and continuously collects vital signs, accelerometer data from a special wearable (such as a smartwatch), and visual data in patient video clips collected from cameras. This data is processed by an anomaly detection module, which includes a fall detection model and algorithms to identify and alert caregivers of the patient's emergency conditions. A distinctive feature of our proposed system is the natural language processing component, developed with MLLMs capable of detecting and recognizing a patient's activity and emotion while responding to healthcare worker's inquiries. Additionally, prompt engineering is employed to integrate all patient information seamlessly. As a result, doctors and nurses can access real-time vital signs and the patient's current state and mood by interacting with an intelligent agent through a user-friendly web application. Our experiments demonstrate that our system is implementable and scalable for real-life scenarios, potentially reducing the workload of medical professionals and healthcare costs. A full-fledged prototype illustrating the functionalities of the system has been developed and being tested to demonstrate the robustness of its various capabilities.

Figures (5)

• Figure 1: Proposed Internet of Things System.
• Figure 2: Comparison of Original and Preprocessed Fall Activity from FallAllD Dataset.
• Figure 3: Natural Language Processing Engine.
• Figure 4: CNN-LSTM Traning vs. Validation Accuracy on the Preprocessed FallAllD dataset.
• Figure 5: CNN-LSTM Model: Confusion Matrix for Test Data on the Preprocessed FallAllD Dataset.