Multi-Modal AI for Remote Patient Monitoring in Cancer Care

TL;DR

This paper investigates remote patient monitoring for cancer patients on systemic therapy, targeting the between-visit period where symptoms go unchecked. It introduces a multi-modal AI framework using a token-based transformer to fuse HALO-X wearable data, QoR-15 surveys, and clinical events and to forecast adverse events within rolling 4-week windows, addressing asynchronous sampling and MNAR. In a prospective observational trial with 50 patients and over 2.1 million data points across thousands of patient-days, the approach achieved AUROC 0.70 and 83.9% accuracy, with prior chemotherapy, A&E visits, wellness check-ins, and daily maximum heart rate identified as key predictors. A case study demonstrates early risk escalation before an event, supporting proactive interventions, and the work establishes feasibility and a blueprint for broader validation and clinical integration.

Abstract

For patients undergoing systemic cancer therapy, the time between clinic visits is full of uncertainties and risks of unmonitored side effects. To bridge this gap in care, we developed and prospectively trialed a multi-modal AI framework for remote patient monitoring (RPM). This system integrates multi-modal data from the HALO-X platform, such as demographics, wearable sensors, daily surveys, and clinical events. Our observational trial is one of the largest of its kind and has collected over 2.1 million data points (6,080 patient-days) of monitoring from 84 patients. We developed and adapted a multi-modal AI model to handle the asynchronous and incomplete nature of real-world RPM data, forecasting a continuous risk of future adverse events. The model achieved an accuracy of 83.9% (AUROC=0.70). Notably, the model identified previous treatments, wellness check-ins, and daily maximum heart rate as key predictive features. A case study demonstrated the model's ability to provide early warnings by outputting escalating risk profiles prior to the event. This work establishes the feasibility of multi-modal AI RPM for cancer care and offers a path toward more proactive patient support.(Accepted at Europe NeurIPS 2025 Multimodal Representation Learning for Healthcare Workshop)

Figures (5)

• Figure 1: Overview of trial design. Multimodal RPM data collection up to 12 months
• Figure 2: Representative patient timeline illustrating longitudinal asynchronous multi-modalities data with missingness. Raw sensor data were aggregated to daily values to avoid clustering in this visualisation. Treatnm.=Treatment; Readm.=Re-admission; Compl.=Complications.
• Figure 3: Model architecture and training pipeline. (a) Value and type of each observation were extracted and tokenized, CVE=Continuous Variable Encoder; (b) Transformer layers for temporal modelling. Static features were encoded via a separate feed-forward neural network; (c) Model uses data before cut-off to generate a binary classification forecast; (d) Evaluation paradigm.
• Figure 4: ROC curve of the model with 95% CI.
• Figure 5: (a) Overview of the most important features, orange bars indicate event features which are less frequent but impactful. Blue bars are remote monitoring feature which happens more regularly and indicates recovery status; (b) Example patient trajectory over time with clinical events annotated. The risk score (y-axis) ranges from 0 to 1, where 1 indicates extremely high risk. Treatm. d./Dose Red.=Treatment Delay/Dose Reduction; A&E=Accident & Emergency department; GP=General Practitioner; QoR-15=Quality of Recovery 15 survey.