BiomechGPT: Towards a Biomechanically Fluent Multimodal Foundation Model for Clinically Relevant Motion Tasks

TL;DR

BiomechGPT introduces a biomechanically fluent multimodal foundation model that unifies analysis of clinically relevant movement tasks through motion-tokenized biomechanical trajectories and language-based reasoning. By coupling a VQ-VAE-based motion tokenizer with a 4-bit QLoRA-finetuned language model trained on 27k prompt–response QA pairs, the approach achieves strong performance across activity recognition, impairment and diagnosis classification, assistive-device identification, and gait metric estimation, outperforming a non-LLM baseline. The work demonstrates positive transfer learning across tasks and shows model-size related performance gains, while acknowledging limitations from dataset size and task coverage. This framework progresses toward accessible, staff-friendly rehabilitation analytics and paves the way for a foundation model that generalizes to home and clinic settings with markerless motion capture data.

Abstract

Advances in markerless motion capture are expanding access to biomechanical movement analysis, making it feasible to obtain high-quality movement data from outpatient clinics, inpatient hospitals, therapy, and even home. Expanding access to movement data in these diverse contexts makes the challenge of performing downstream analytics all the more acute. Creating separate bespoke analysis code for all the tasks end users might want is both intractable and does not take advantage of the common features of human movement underlying them all. Recent studies have shown that fine-tuning language models to accept tokenized movement as an additional modality enables successful descriptive captioning of movement. Here, we explore whether such a multimodal motion-language model can answer detailed, clinically meaningful questions about movement. We collected over 30 hours of biomechanics from nearly 500 participants, many with movement impairments from a variety of etiologies, performing a range of movements used in clinical outcomes assessments. After tokenizing these movement trajectories, we created a multimodal dataset of motion-related questions and answers spanning a range of tasks. We developed BiomechGPT, a multimodal biomechanics-language model, on this dataset. Our results show that BiomechGPT demonstrates high performance across a range of tasks such as activity recognition, identifying movement impairments, diagnosis, scoring clinical outcomes, and measuring walking. BiomechGPT provides an important step towards a foundation model for rehabilitation movement data.

Figures (4)

• Figure 1: Screenshot from our chat interface, which allows selecting a trial from our motion dataset and chatting about that motion trajectory.
• Figure 2: Confusion matrix for activity classification from one of our 4B models trained on all of the tasks.
• Figure 3: Confusion matrix for impaired movement classification (top left) and diagnosis classification (top right), assistive device use (bottom left) and history of falls (bottom right) from one of our 4B BiomechGPT models trained on all of the tasks.
• Figure 4: Regression for walking cadence (left) and and speed (right) from a 4B parameter model trained on all tasks.