CardioAI: A Multimodal AI-based System to Support Symptom Monitoring and Risk Detection of Cancer Treatment-Induced Cardiotoxicity

TL;DR

This work tackles the challenge of early detection of cancer treatment-induced cardiotoxicity while addressing clinician workload by proposing CardioAI, a multimodal AI system that combines wearable sensor data with an LLM-based voice assistant and an explainable risk-prediction module. The design process is grounded in participatory design with 11 clinicians and evaluated via a heuristic study with 4 clinicians, showing the system can be integrated into clinical workflows, reduce information overload, and support proactive decisions. CardioAI’s backend comprises a RAG-powered conversation module, a Transformer-based risk predictor with SHAP explanations, and a summarization component that generates daily patient summaries; the frontend presents five clinician-facing modules, enabling seamless viewing of patient information, biosignals, AI risk scores, and conversation logs. The study highlights practical benefits and design considerations for deploying multimodal AI in healthcare, while noting limitations such as small sample size, prototype status, and the need for EHR integration and patient-facing evaluations to validate real-world utility and trust.”

Abstract

Despite recent advances in cancer treatments that prolong patients' lives, treatment-induced cardiotoxicity remains one severe side effect. The clinical decision-making of cardiotoxicity is challenging, as non-clinical symptoms can be missed until life-threatening events occur at a later stage, and clinicians already have a high workload centered on the treatment, not the side effects. Our project starts with a participatory design study with 11 clinicians to understand their practices and needs; then we build a multimodal AI system, CardioAI, that integrates wearables and LLM-powered voice assistants to monitor multimodal non-clinical symptoms. Also, the system includes an explainable risk prediction module that can generate cardiotoxicity risk scores and summaries as explanations to support clinicians' decision-making. We conducted a heuristic evaluation with four clinical experts and found that they all believe CardioAI integrates well into their workflow, reduces their information overload, and enables them to make more informed decisions.

Figures (4)

• Figure 1: Our participatory design session. A participant is suggesting design revisions on the initial UI.
• Figure 2: Examples of low-fidelity visualization options for heart rate data used to elicit clinician feedback.
• Figure 3: AI-based Cardiotoxicity Risk Score Prediction Framework.
• Figure 4: System Architecture of CardioAI. It integrates a wearable and a smart speaker to continuously collect physiological data and patient-reported symptoms. Data is processed by three key LLM-powered backend components: a conversation module, a summarization module, and a risk prediction module. The processed data, including key health summaries and cardiotoxicity risk scores with explainability, is then visualized on a clinician-facing dashboard.