VL-OrdinalFormer: Vision Language Guided Ordinal Transformers for Interpretable Knee Osteoarthritis Grading

TL;DR

This paper tackles automated knee osteoarthritis grading by KL scale, focusing on the challenging boundary between early grades KL1 and KL2. It introduces VL-OrdinalFormer, a vision-language guided ordinal classifier that combines a ViT backbone with CORAL-based ordinal regression and a CLIP-driven semantic supervision to ground visual features in clinically meaningful textual concepts. The approach uses strike-accurate training strategies (class-aware weighting, cross-validation, TTA, and threshold tuning) and demonstrates state-of-the-art performance on the OAI knee dataset, with strong improvements for mid grades and robust interpretability via Grad-CAM and CLIP heatmaps. The work suggests that integrating ordinal structure with vision-language representations enhances both accuracy and clinical interpretability, supporting reliable KOA grading and disease progression assessment in routine radiology workflows.

Abstract

Knee osteoarthritis (KOA) is a leading cause of disability worldwide, and accurate severity assessment using the Kellgren Lawrence (KL) grading system is critical for clinical decision making. However, radiographic distinctions between early disease stages, particularly KL1 and KL2, are subtle and frequently lead to inter-observer variability among radiologists. To address these challenges, we propose VLOrdinalFormer, a vision language guided ordinal learning framework for fully automated KOA grading from knee radiographs. The proposed method combines a ViT L16 backbone with CORAL based ordinal regression and a Contrastive Language Image Pretraining (CLIP) driven semantic alignment module, allowing the model to incorporate clinically meaningful textual concepts related to joint space narrowing, osteophyte formation, and subchondral sclerosis. To improve robustness and mitigate overfitting, we employ stratified five fold cross validation, class aware re weighting to emphasize challenging intermediate grades, and test time augmentation with global threshold optimization. Experiments conducted on the publicly available OAI kneeKL224 dataset demonstrate that VLOrdinalFormer achieves state of the art performance, outperforming CNN and ViT baselines in terms of macro F1 score and overall accuracy. Notably, the proposed framework yields substantial performance gains for KL1 and KL2 without compromising classification accuracy for mild or severe cases. In addition, interpretability analyses using Grad CAM and CLIP similarity maps confirm that the model consistently attends to clinically relevant anatomical regions. These results highlight the potential of vision language aligned ordinal transformers as reliable and interpretable tools for KOA grading and disease progression assessment in routine radiological practice.

Figures (6)

• Figure 1: Knee joint samples of all KL grades and their corresponding criterion.
• Figure 2: KL grade distribution in the OAI knee radiograph dataset.
• Figure 3: Illustration of the preprocessing pipeline. From left to right: (a) original knee crop, (b) YOLO-like localization of the diagnostic region of interest, and (c) final preprocessed 224$\times$224 patch used as input to the ViT-CORAL model.
• Figure 4: Schematic overview of the proposed vision-language-guided framework for automatic KOA severity prediction. The input knee X-ray image is processed through a ViT-L/16 backbone with 16×16 patch embedding. Feature representations from the [CLS] token are passed through a CORAL ordinal regression head to predict KL grades (0–4). An optional VLM distillation module aligns visual features with clinical text embeddings from CLIP. Finally, attention rollout visualizations highlight the anatomical regions influencing the model’s decisions.
• Figure 5: Confusion matrix.