Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression

V. K. Cody Bumgardner, Mitchell A. Klusty, Mahmut S. Gokmen, Evan W. Damron

Abstract

Automated radiology report generation from 3D computed tomography (CT) volumes is challenging due to extreme sequence lengths, severe class imbalance, and the tendency of large language models (LLMs) to ignore visual tokens in favor of linguistic priors. We present Ker-VLJEPA-3B, a four-phase curriculum learning framework for free-text report generation from thoracic CT volumes. A phased training curriculum progressively adapts a Llama 3.2 3B decoder to ground its output in visual features from a frozen, self-supervised encoder. Our visual backbone (LeJEPA ViT-Large) is trained via self-supervised joint-embedding prediction on unlabeled CTs, without text supervision. Unlike contrastive models (CLIP, BiomedCLIP), this language-free backbone yields modality-pure representations. Vision-language alignment is deferred to the curriculum's bridge and generation phases. This modality-agnostic design can integrate any self-supervised encoder into an LLM without paired text during foundation training. Methodological innovations include: (1) zone-constrained cross-attention compressing slice embeddings into 32 spatially-grounded visual tokens; (2) PCA whitening of anisotropic LLM embeddings; (3) a positive-findings-only strategy eliminating posterior collapse; (4) warm bridge initialization transferring projection weights; and (5) selective cross-attention freezing with elastic weight consolidation to prevent catastrophic forgetting. Evaluated on the CT-RATE benchmark (2,984 validation volumes, 18 classes), Ker-VLJEPA-3B achieves a macro F1 of 0.429, surpassing the state-of-the-art (U-VLM, macro F1 = 0.414) by 3.6%, and reaching 0.448 (+8.2%) with threshold optimization. Ablation studies confirm 56.6% of generation quality derives from patient-specific visual content. Code and weights are available.

Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression

Abstract

Automated radiology report generation from 3D computed tomography (CT) volumes is challenging due to extreme sequence lengths, severe class imbalance, and the tendency of large language models (LLMs) to ignore visual tokens in favor of linguistic priors. We present Ker-VLJEPA-3B, a four-phase curriculum learning framework for free-text report generation from thoracic CT volumes. A phased training curriculum progressively adapts a Llama 3.2 3B decoder to ground its output in visual features from a frozen, self-supervised encoder. Our visual backbone (LeJEPA ViT-Large) is trained via self-supervised joint-embedding prediction on unlabeled CTs, without text supervision. Unlike contrastive models (CLIP, BiomedCLIP), this language-free backbone yields modality-pure representations. Vision-language alignment is deferred to the curriculum's bridge and generation phases. This modality-agnostic design can integrate any self-supervised encoder into an LLM without paired text during foundation training. Methodological innovations include: (1) zone-constrained cross-attention compressing slice embeddings into 32 spatially-grounded visual tokens; (2) PCA whitening of anisotropic LLM embeddings; (3) a positive-findings-only strategy eliminating posterior collapse; (4) warm bridge initialization transferring projection weights; and (5) selective cross-attention freezing with elastic weight consolidation to prevent catastrophic forgetting. Evaluated on the CT-RATE benchmark (2,984 validation volumes, 18 classes), Ker-VLJEPA-3B achieves a macro F1 of 0.429, surpassing the state-of-the-art (U-VLM, macro F1 = 0.414) by 3.6%, and reaching 0.448 (+8.2%) with threshold optimization. Ablation studies confirm 56.6% of generation quality derives from patient-specific visual content. Code and weights are available.
Paper Structure (38 sections, 13 equations, 2 figures, 11 tables)

This paper contains 38 sections, 13 equations, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Models in Medical Imaging
  4. 3D CT Understanding and Report Generation
  5. Embedding Grafting and Vision-Language JEPAs
  6. Curriculum Learning and Continual Learning
  7. Methods
  8. Problem Formulation
  9. Architecture Overview
  10. Visual Encoder: Zone-Constrained Cross-Attention
  11. JEPA Predictor and Norm Calibration
  12. Embedding Grafting and Multi-Layer Injection
  13. JEPA Embedding Head and PCA Whitening
  14. Four-Phase Curriculum Learning
  15. Phase 1: Visual Alignment (Classification-Driven)
  16. ...and 23 more sections

Figures (2)

  • Figure 1: Overview of the Ker-VLJEPA-3B architecture. CT slice embeddings from a frozen LeJEPA ViT-Large are compressed into 32 spatially-grounded tokens via zone-constrained cross-attention, projected to LLM space via the JEPA predictor with norm calibration, and grafted into Llama 3.2 3B at both the embedding level and intermediate layers (7, 14, 21) through gated cross-attention adapters. Auxiliary branches provide supervision via the JEPA embedding head (256-d whitened space) and an 18-class classifier.
  • Figure 2: Four-phase curriculum learning pipeline. Each phase progressively builds capability from visual discrimination (Phase 1) through contrastive alignment (Phase 2) to free-text generation (Phase 3) and narrative adaptation (Phase 4). Colored boxes indicate trainable components; dashed boxes indicate frozen components. The warm bridge initialization (red dashed arrow) transfers 27 converged bridge components and 392 LoRA tensors from a prior Phase 3 run, providing immediate convergence and eliminating the cold-start problem.