Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMTA: Multi Membership Temporal Attention for Fine-Grained Stroke Rehabilitation Assessment

Halil Ismail Helvaci, Justin Huber, Jihye Bae, Sen-ching Samson Cheung

TL;DR

Multi-Membership Temporal Attention (MMTA), a high-resolution temporal transformer for fine-grained rehabilitation assessment that supports both video and wearable IMU inputs within a unified single-stage architecture, making it applicable to both clinical and home settings.

Abstract

To empower the iterative assessments involved during a person's rehabilitation, automated assessment of a person's abilities during daily activities requires temporally precise segmentation of fine-grained actions in therapy videos. Existing temporal action segmentation (TAS) models struggle to capture sub-second micro-movements while retaining exercise context, blurring rapid phase transitions and limiting reliable downstream assessment of motor recovery. We introduce Multi-Membership Temporal Attention (MMTA), a high-resolution temporal transformer for fine-grained rehabilitation assessment. Unlike standard temporal attention, which assigns each frame a single attention context per layer, MMTA lets each frame attend to multiple locally normalized temporal attention windows within the same layer. We fuse these concurrent temporal views via feature-space overlap resolution, preserving competing local contexts near transitions while enabling longer-range reasoning through layer-wise propagation. This increases boundary sensitivity without additional depth or multi-stage refinement. MMTA supports both video and wearable IMU inputs within a unified single-stage architecture, making it applicable to both clinical and home settings. MMTA consistently improves over the Global Attention transformer, boosting Edit Score by +1.3 (Video) and +1.6 (IMU) on StrokeRehab while further improving 50Salads by +3.3. Ablations confirm that performance gains stem from multi-membership temporal views rather than architectural complexity, offering a practical solution for resource-constrained rehabilitation assessment.

MMTA: Multi Membership Temporal Attention for Fine-Grained Stroke Rehabilitation Assessment

TL;DR

Multi-Membership Temporal Attention (MMTA), a high-resolution temporal transformer for fine-grained rehabilitation assessment that supports both video and wearable IMU inputs within a unified single-stage architecture, making it applicable to both clinical and home settings.

Abstract

To empower the iterative assessments involved during a person's rehabilitation, automated assessment of a person's abilities during daily activities requires temporally precise segmentation of fine-grained actions in therapy videos. Existing temporal action segmentation (TAS) models struggle to capture sub-second micro-movements while retaining exercise context, blurring rapid phase transitions and limiting reliable downstream assessment of motor recovery. We introduce Multi-Membership Temporal Attention (MMTA), a high-resolution temporal transformer for fine-grained rehabilitation assessment. Unlike standard temporal attention, which assigns each frame a single attention context per layer, MMTA lets each frame attend to multiple locally normalized temporal attention windows within the same layer. We fuse these concurrent temporal views via feature-space overlap resolution, preserving competing local contexts near transitions while enabling longer-range reasoning through layer-wise propagation. This increases boundary sensitivity without additional depth or multi-stage refinement. MMTA supports both video and wearable IMU inputs within a unified single-stage architecture, making it applicable to both clinical and home settings. MMTA consistently improves over the Global Attention transformer, boosting Edit Score by +1.3 (Video) and +1.6 (IMU) on StrokeRehab while further improving 50Salads by +3.3. Ablations confirm that performance gains stem from multi-membership temporal views rather than architectural complexity, offering a practical solution for resource-constrained rehabilitation assessment.
Paper Structure (6 sections, 5 equations, 2 figures, 3 tables)

This paper contains 6 sections, 5 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Experiments
  4. Conclusion

Figures (2)

  • Figure 1: MMTA replaces global attention with locally normalized attention over overlapping temporal windows. Frames in overlap regions receive multiple window-conditioned updates, reconciled via overlap-resolution aggregation.
  • Figure 2: Qualitative comparison of predicted (Pred) and ground-truth (GT) action labels on an example test sequence for IMU data (top) and video data (bottom).