Diagnosing Shoulder Disorders Using Multimodal Large Language Models and Consumer-Grade Cameras

TL;DR

This work tackles scalable, low‑cost screening of shoulder disorders in resource‑limited settings by leveraging consumer‑grade video data and Multimodal LLMs. It introduces HMVDx, a two‑stage framework that splits action understanding (via Gemini‑1.5‑Pro) and disease reasoning (via DeepSeek‑R1), guided by a Motion Trajectories Prompt Framework and three prompts (Prompt‑A/B/C) for robust video‑to‑diagnosis workflows. The authors also propose a novel Usability Index (UI) to evaluate the entire diagnostic pathway, emphasizing action recognition integrity, rationality of judgments, and final accuracy. Empirical results show HMVDx yields up to a 79.6% accuracy gain over direct video diagnosis and achieves the best overall usability, though Scenario 3 reveals remaining gaps under strict full‑process constraints. The work points to future enhancements via SFT/RAG/knowledge graphs, multilingual data, and larger, more diverse datasets to broaden applicability in primary care.

Abstract

Shoulder disorders, such as frozen shoulder (a.k.a., adhesive capsulitis), are common conditions affecting the health of people worldwide, and have a high incidence rate among the elderly and workers engaged in repetitive shoulder tasks. In regions with scarce medical resources, achieving early and accurate diagnosis poses significant challenges, and there is an urgent need for low-cost and easily scalable auxiliary diagnostic solutions. This research introduces videos captured by consumer-grade devices as the basis for diagnosis, reducing the cost for users. We focus on the innovative application of Multimodal Large Language Models (MLLMs) in the preliminary diagnosis of shoulder disorders and propose a Hybrid Motion Video Diagnosis framework (HMVDx). This framework divides the two tasks of action understanding and disease diagnosis, which are respectively completed by two MLLMs. In addition to traditional evaluation indicators, this work proposes a novel metric called Usability Index by the logical process of medical decision-making (action recognition, movement diagnosis, and final diagnosis). This index evaluates the effectiveness of MLLMs in the medical field from the perspective of the entire medical diagnostic pathway, revealing the potential value of low-cost MLLMs in medical applications for medical practitioners. In experimental comparisons, the accuracy of HMVDx in diagnosing shoulder joint injuries has increased by 79.6\% compared with direct video diagnosis, a significant technical contribution to future research on the application of MLLMs for video understanding in the medical field.

Figures (5)

• Figure 1: (a) Diagnostic Process Comparison. In the traditional diagnostic process, patients must visit a hospital to determine their health status. In contrast, our envisioned AI-powered diagnostic process enables patients to assess their condition remotely using our Video Diagnosis AI Assistant, without the need to visit a hospital. This greatly reduces patient wait times and alleviates the burden on healthcare systems. On the right, we illustrate an application scenario of our framework: a patient performs specific movements, and our model analyzes the motion video to provide a diagnosis. (b) Data Processing Pipeline. Our data processing pipeline is designed with both personal privacy protection and engineering optimization in mind for handling human motion video data.
• Figure 2: Pie charts illustrating the distribution of disease status, age groups, and gender in our dataset.
• Figure 3: Overview of Framework Variants for Video Diagnosis
• Figure 4: Comparison of different methods and model sizes in terms of Accuracy, F1 Score, and UI.
• Figure 5: Qualitative Results. Our method demonstrates superior performance over GPT-4o and DVDx