Morphology-Aware KOA Classification: Integrating Graph Priors with Vision Models

TL;DR

This work tackles KOA severity scoring from radiographs by introducing a geometry-aware multimodal framework that combines SAM-derived joint morphology graphs with a vision encoder. A MorphoGraph graph encoder provides a morphology-rich embedding, while a vision backbone supplies complementary radiographic features; these are aligned via mutual information maximization and a learnable cross-modal translator to produce a joint KOA representation. On the Osteoarthritis Initiative dataset, the approach outperforms single-modality baselines by up to 10 percentage points (nearly 80% accuracy) and surpasses prior state-of-the-art by about 8% in accuracy and 11% in F1. The results demonstrate that incorporating explicit anatomical priors in radiographic analysis yields more accurate and robust KOA grading, with potential clinical impact for faster, more reliable OA assessment.

Abstract

Knee osteoarthritis (KOA) diagnosis from radiographs remains challenging due to the subtle morphological details that standard deep learning models struggle to capture effectively. We propose a novel multimodal framework that combines anatomical structure with radiographic features by integrating a morphological graph representation - derived from Segment Anything Model (SAM) segmentations - with a vision encoder. Our approach enforces alignment between geometry-informed graph embeddings and radiographic features through mutual information maximization, significantly improving KOA classification accuracy. By constructing graphs from anatomical features, we introduce explicit morphological priors that mirror clinical assessment criteria, enriching the feature space and enhancing the model's inductive bias. Experiments on the Osteoarthritis Initiative dataset demonstrate that our approach surpasses single-modality baselines by up to 10\% in accuracy (reaching nearly 80\%), while outperforming existing state-of-the-art methods by 8\% in accuracy and 11\% in F1 score. These results underscore the critical importance of incorporating anatomical structure into radiographic analysis for accurate KOA severity grading.

Figures (2)

• Figure 1: Overview of our pipeline: First, SAM is prompted to generate candidate masks $\mathcal{M}$. The best mask $m^*$ is chosen based on IoU with upper and lower bone templates $(T_U, T_L)$. Next, a morphological graph $G_{\mathrm{joint}}$ is constructed from the joint boundary and processed by the graph encoder $f_{\mathrm{graph}}(\cdot;\Theta)$. Simultaneously, the radiograph is passed through a vision encoder $f_{\mathrm{vision}}(\cdot;\Theta)$. The translation head $T_{\mathrm{head}}(z_{\mathrm{vision}};\Theta)$ aligns the vision embedding with the graph domain, and the fusion module $f_{\mathrm{merge}}([\;z_{\mathrm{graph}},\,z_{\mathrm{trans}}];\Theta)$ combines both representations. Both $T_{\mathrm{head}}$ and $f_{\mathrm{merge}}$ are linear multi perceptron layers. Finally, a linear layer produces the KOA severity logits $\hat{y}$.
• Figure 2: T-SNE visualizations of learned embeddings of MM-F (MIM) ViT Large across four configurations. From left to right: (1) Vision-only model, (2) classical multimodal fusion, (3) multimodal fusion with mutual information maximization (MIM), and (4) graph-only representation. Colors indicate KOA severity classes.