A Fine-Grained Attention and Geometric Correspondence Model for Musculoskeletal Risk Classification in Athletes Using Multimodal Visual and Skeletal Features

TL;DR

This work tackles musculoskeletal risk assessment in athletes by introducing ViSK-GAT, a multimodal Visual-Skeletal Geometric Attention Transformer that fuses image and skeletal coordinate data. It builds the MusDis-Sports dataset, combining sports images with MediaPipe 33-point skeletal keypoints and REBA-based eight-class risk labels, to train and evaluate the model. ViSK-GAT employs a Fine-Grained Attention Module (FGAM) and a Multimodal Geometric Correspondence Module (MGCM) to achieve superior cross-modal alignment, yielding a test accuracy of 93.89% and MCC/Cohen's kappa of 0.93, outperforming nine transfer-learning baselines. The approach enables robust, real-time posture risk classification in dynamic athletic settings and sets the stage for richer metadata integration and broader ergonomic applications.

Abstract

Musculoskeletal disorders pose significant risks to athletes, and assessing risk early is important for prevention. However, most existing methods are designed for controlled settings and fail to reliably assess risk in complex environments due to their reliance on a single type of data. This research introduces ViSK-GAT (Visual-Skeletal Geometric Attention Transformer), a novel multimodal deep learning framework that classifies musculoskeletal risk using both visual and skeletal coordinate-based features. A custom multimodal dataset (MusDis-Sports) was created by combining images and skeletal coordinates, with each sample labeled into eight risk categories based on the Rapid Entire Body Assessment (REBA) system. ViSK-GAT integrates two innovative modules: the Fine-Grained Attention Module (FGAM), which refines inter-modal features via cross-attention between visual and skeletal inputs, and the Multimodal Geometric Correspondence Module (MGCM), which enhances cross-modal alignment between image features and coordinates. The model achieved robust performance, with all key metrics exceeding 93%. Regression results also indicated a low RMSE of 0.1205 and MAE of 0.0156. ViSK-GAT consistently outperformed nine popular transfer learning backbones and showed its potential to advance AI-driven musculoskeletal risk assessment and enable early, impactful interventions in sports.

Figures (6)

• Figure 1: We processed the img feature with Fine-Grained Attention Module (FGMA) and used a coordinate feature encoder for skeletal feature processing. The Multimodal Geometric Correspondence Module (MGCM) uses a cross-attention mechanism for the image and coordinate feature integration and risk assessment.
• Figure 2: Sample images from the constructed MusDis-Sports dataset showing athletes in various sports-related postures with overlaid 2D skeletal keypoints extracted using the MediaPipe framework. These samples are used for training and evaluating musculoskeletal risk classification.
• Figure 3: Overview of the proposed multimodal framework (ViSK-GAT) for ergonomic posture risk assessment. The system integrates visual features and skeletal coordinates through a hybrid backbone, enhanced by a Fine-Grained Attention Module and a Multimodal Geometric Correspondence Module.
• Figure 4: Architecture of the proposed Fine-Grained Attention Module for enhancing token representations.
• Figure 5: Proposed architecture of the Multimodal Geometric Correspondence Module (MGCM). This is a cross-attention transformer designed to bridge the gap between image tokens (Query) and skeletal coordinate embeddings (Key and Value). The process involves: (1) projecting both features into a common latent space; (2) using cross-attention to align visual appearance with structural posture information; (3) concatenating the attention output with the image feature; and (4) refining the combined feature using stacked Transformer layers to produce the final Fused Feature Vector ($F_{corr}$) for risk classification.