Automated ARAT Scoring Using Multimodal Video Analysis, Multi-View Fusion, and Hierarchical Bayesian Models: A Clinician Study
Tamim Ahmed, Thanassis Rikakis
TL;DR
This paper tackles the challenge of time-consuming and variable manual ARAT scoring in stroke rehabilitation. It introduces a multimodal framework that fuses SlowFast, I3D, and TimeSformer pipelines with OpenPose keypoints and object data across multiple views, augmented by Hierarchical Bayesian Models for interpretable movement-quality inferences. A clinician dashboard consolidates task scores, execution times, and impairment diagnostics, with a clinical study (n=5 clinicians) analyzing 500 automated video ratings. The approach achieves 89.0% validation accuracy with late fusion and 91.0% ARAT score agreement, while HBMs align with manual assessments, suggesting practical, scalable impact for automated, interpretable ARAT scoring in clinical settings.
Abstract
Manual scoring of the Action Research Arm Test (ARAT) for upper extremity assessment in stroke rehabilitation is time-intensive and variable. We propose an automated ARAT scoring system integrating multimodal video analysis with SlowFast, I3D, and Transformer-based models using OpenPose keypoints and object locations. Our approach employs multi-view data (ipsilateral, contralateral, and top perspectives), applying early and late fusion to combine features across views and models. Hierarchical Bayesian Models (HBMs) infer movement quality components, enhancing interpretability. A clinician dashboard displays task scores, execution times, and quality assessments. We conducted a study with five clinicians who reviewed 500 video ratings generated by our system, providing feedback on its accuracy and usability. Evaluated on a stroke rehabilitation dataset, our framework achieves 89.0% validation accuracy with late fusion, with HBMs aligning closely with manual assessments. This work advances automated rehabilitation by offering a scalable, interpretable solution with clinical validation.