Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Digital FAST: An AI-Driven Multimodal Framework for Rapid and Early Stroke Screening

Ngoc-Khai Hoang, Thi-Nhu-Mai Nguyen, Huy-Hieu Pham

TL;DR

The paper addresses rapid, non-invasive prehospital stroke screening by integrating facial landmark dynamics, speech signals, and upper-body movements captured during the F.A.S.T. assessment. It introduces a multimodal architecture with Transformer-based facial and audio encoders and an MLP-Mixer pose encoder, fused via an attention mechanism into a binary stroke classifier. Experiments on a self-collected dataset show multimodal models outperform unimodal baselines, with a fine-tuned fusion achieving sensitivity 1.0 and an AUC of 1.0, while maintaining high specificity. The work demonstrates the feasibility of multimodal, transfer-learning based screening in prehospital settings, and highlights the need for larger, clinically representative datasets for real-world deployment.

Abstract

Early identification of stroke symptoms is essential for enabling timely intervention and improving patient outcomes, particularly in prehospital settings. This study presents a fast, non-invasive multimodal deep learning framework for automatic binary stroke screening based on data collected during the F.A.S.T. assessment. The proposed approach integrates complementary information from facial expressions, speech signals, and upper-body movements to enhance diagnostic robustness. Facial dynamics are represented using landmark based features and modeled with a Transformer architecture to capture temporal dependencies. Speech signals are converted into mel spectrograms and processed using an Audio Spectrogram Transformer, while upper-body pose sequences are analyzed with an MLP-Mixer network to model spatiotemporal motion patterns. The extracted modality specific representations are combined through an attention-based fusion mechanism to effectively learn cross modal interactions. Experiments conducted on a self-collected dataset of 222 videos from 37 subjects demonstrate that the proposed multimodal model consistently outperforms unimodal baselines, achieving 95.83% accuracy and a 96.00% F1-score. The model attains a strong balance between sensitivity and specificity and successfully detects all stroke cases in the test set. These results highlight the potential of multimodal learning and transfer learning for early stroke screening, while emphasizing the need for larger, clinically representative datasets to support reliable real-world deployment.

Digital FAST: An AI-Driven Multimodal Framework for Rapid and Early Stroke Screening

TL;DR

The paper addresses rapid, non-invasive prehospital stroke screening by integrating facial landmark dynamics, speech signals, and upper-body movements captured during the F.A.S.T. assessment. It introduces a multimodal architecture with Transformer-based facial and audio encoders and an MLP-Mixer pose encoder, fused via an attention mechanism into a binary stroke classifier. Experiments on a self-collected dataset show multimodal models outperform unimodal baselines, with a fine-tuned fusion achieving sensitivity 1.0 and an AUC of 1.0, while maintaining high specificity. The work demonstrates the feasibility of multimodal, transfer-learning based screening in prehospital settings, and highlights the need for larger, clinically representative datasets for real-world deployment.

Abstract

Early identification of stroke symptoms is essential for enabling timely intervention and improving patient outcomes, particularly in prehospital settings. This study presents a fast, non-invasive multimodal deep learning framework for automatic binary stroke screening based on data collected during the F.A.S.T. assessment. The proposed approach integrates complementary information from facial expressions, speech signals, and upper-body movements to enhance diagnostic robustness. Facial dynamics are represented using landmark based features and modeled with a Transformer architecture to capture temporal dependencies. Speech signals are converted into mel spectrograms and processed using an Audio Spectrogram Transformer, while upper-body pose sequences are analyzed with an MLP-Mixer network to model spatiotemporal motion patterns. The extracted modality specific representations are combined through an attention-based fusion mechanism to effectively learn cross modal interactions. Experiments conducted on a self-collected dataset of 222 videos from 37 subjects demonstrate that the proposed multimodal model consistently outperforms unimodal baselines, achieving 95.83% accuracy and a 96.00% F1-score. The model attains a strong balance between sensitivity and specificity and successfully detects all stroke cases in the test set. These results highlight the potential of multimodal learning and transfer learning for early stroke screening, while emphasizing the need for larger, clinically representative datasets to support reliable real-world deployment.
Paper Structure (13 sections, 3 figures, 4 tables)

This paper contains 13 sections, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Proposed Framework
  5. Dataset
  6. Data acquisition
  7. Data preprocessing
  8. Experiment
  9. Implementation Details
  10. Experimental Result
  11. Ablation Study
  12. Dicussion
  13. Conclusion

Figures (3)

  • Figure 1: The proposed framework comprises four stages: (a) Raw Input Processing, which transforms raw visual and audio data into structured representations; (b) Modality Encoders, which extract high-level embeddings using modality-specific models; (c) Multimodal Fusion, which integrates the embeddings through a learnable fusion mechanism; and (d) Classification Head, which produces the final binary stroke prediction..
  • Figure 2: Illustrative examples of facial and pose landmarks adapted from the Google MediaPipe documentation.
  • Figure 3: Side-by-side comparison of mel spectrograms from (a) a healthy subject and (b) a stroke-simulated subject, illustrating differences in temporal and spectral speech patterns at 80 mel bins.