Universal Skeleton Understanding via Differentiable Rendering and MLLMs

Abstract

Multimodal large language models (MLLMs) exhibit strong visual-language reasoning, yet remain confined to their native modalities and cannot directly process structured, non-visual data such as human skeletons. Existing methods either compress skeleton dynamics into lossy feature vectors for text alignment, or quantize motion into discrete tokens that generalize poorly across heterogeneous skeleton formats. We present SkeletonLLM, which achieves universal skeleton understanding by translating arbitrary skeleton sequences into the MLLM's native visual modality. At its core is DrAction, a differentiable, format-agnostic renderer that converts skeletal kinematics into compact image sequences. Because the pipeline is end-to-end differentiable, MLLM gradients can directly guide the rendering to produce task-informative visual tokens. To further enhance reasoning capabilities, we introduce a cooperative training strategy: Causal Reasoning Distillation transfers structured, step-by-step reasoning from a teacher model, while Discriminative Finetuning sharpens decision boundaries between confusable actions. SkeletonLLM demonstrates strong generalization on diverse tasks including recognition, captioning, reasoning, and cross-format transfer -- suggesting a viable path for applying MLLMs to non-native modalities. Code will be released upon acceptance.

Figures (15)

• Figure 1: Breaking Format Silos and the Modality Gap.(Top) MLLMs possess strong reasoning capabilities but cannot natively process structured skeleton data. (Middle) Traditional alignment methods are tied to specific skeleton topologies, compressing motion into a single vector for matching against text embeddings, which creates representation bottlenecks and brittle semantics. (Bottom) Our SkeletonLLM uses DrAction, a differentiable renderer that translates a single skeleton sequence of any format into the MLLM's native visual language, enabling end-to-end optimization and unlocking powerful visual reasoning for diverse tasks.
• Figure 2: Overview of SkeletonLLM. The pipeline follows a Render-Reason-Respond process for universal understanding. Given a skeleton sequence, DrAction lifts joint trajectories into deformable 3D Gaussian primitives and renders motion-aware images. Joint transforms are computed via Linear Blend Skinning, and kinematic cues (depth, velocity) are fused through a Neural Feature Modulator. All parameters are optimized end-to-end by gradients from the MLLM. The rendered frames are processed by the MLLM's vision encoder and a projector to yield visual tokens. During training, CR-Distill supervises with teacher-generated causal chains describing body-part dynamics, while Disc-FT sharpens decision boundaries via binary queries over confusing action pairs.
• Figure 3: Cross-format rendering by DrAction. Top row: DrAction renders skeletons from four different formats into visually consistent image sequences. Bottom row: the underlying skeleton topologies vary significantly in joint count and connectivity---NW-UCLA (Kinect v1, 20 joints), NTU (Kinect v2, 25 joints), NTU-2D (pose estimation, 17 joints), and HumanML3D (MoCap, 22 joints). Despite these differences, DrAction produces a unified visual language, enabling seamless cross-format transfer.
• Figure 4: Qualitative comparison of rendering methods. Fixed renderers (3D+Velocity, 2D, JTM jtmacmmm) produce visualizations that are either generic, information-poor, or perceptually complex. DrAction learns an abstract representation. With the NFM, it dynamically highlights kinematically salient regions (e.g., the kicking leg), producing a more informative visual language for the MLLM. A Video Gallery in the Supplementary Material showcases DrAction-rendered videos.
• Figure 5: Our Progressive Training Pipeline. To address the joint optimization challenge, we progressively activate and fine-tune model components. The training curriculum begins with (a) warming up the renderer to generate intelligible visuals and concludes with (d) refining recognition, both utilizing a multiple-choice question & answer (MQA) task. In between, the strategy incorporates (b) learning discriminative features via a binary judgment task and (c) instilling causal reasoning through knowledge distillation from a teacher model.