Fixed-Budget Parameter-Efficient Training with Frozen Encoders Improves Multimodal Chest X-Ray Classification

TL;DR

This work tackles the high computational cost of multimodal chest X-ray classification by evaluating parameter-efficient training (PET) methods that freeze encoders and train a small fusion module under a fixed parameter budget. Across IU Chest X-Ray data with leakage-mitigated reports and external CheXpert validation, PET variants (Frozen, LoRA, BitFit, Adapters) achieve AUROCs of $0.892$–$0.908$ using $2.51\%$ of parameters, far outperforming full fine-tuning at $0.770$ AUROC with all parameters. A parameter attribution analysis reveals that gains arise primarily from allocating parameters to the cross-modal fusion component rather than from true cross-modal synergy, highlighting the efficacy of frozen-encoder designs for efficient deployment. Nonetheless, model calibration remains a challenge, necessitating post-hoc calibration for clinical reliability, while leakage auditing and multi-institution validation are important for real-world adoption.

Abstract

Multimodal chest X-Ray analysis often fine-tunes large vision-language models, which is computationally costly. We study parameter-efficient training (PET) strategies, including frozen encoders, BitFit, LoRA, and adapters for multi-label classification on the Indiana University Chest X-Ray dataset (3,851 image-report pairs; 579 test samples). To mitigate data leakage, we redact pathology terms from reports used as text inputs while retaining clinical context. Under a fixed parameter budget (2.37M parameters, 2.51% of total), all PET variants achieve AUROC between 0.892 and 0.908, outperforming full fine-tuning (0.770 AUROC), which uses 94.3M trainable parameters, a 40x reduction. External validation on CheXpert (224,316 images, 58x larger) confirms scalability: all PET methods achieve >0.69 AUROC with <9% trainable parameters, with Adapter achieving best performance (0.7214 AUROC). Budget-matched comparisons reveal that vision-only models (0.653 AUROC, 1.06M parameters) outperform budget-matched multimodal models (0.641 AUROC, 1.06M parameters), indicating improvements arise primarily from parameter allocation rather than cross-modal synergy. While PET methods show degraded calibration (ECE: 0.29-0.34) compared to simpler models (ECE: 0.049), this represents a tractable limitation addressable through post-hoc calibration methods. These findings demonstrate that frozen encoder strategies provide superior discrimination at substantially reduced computational cost, though calibration correction is essential for clinical deployment.

Figures (6)

• Figure 1: Chest X-Ray images showing diverse pathological conditions from the Indiana University dataset.
• Figure 2: Parameter-Efficient Multimodal Architecture for Chest X-Ray Analysis. Our framework employs frozen ResNet-50 and DistilBERT encoders with a trainable cross-modal fusion module. The architecture processes chest X-Rays and redacted clinical reports through separate encoder pipelines, projects features to a common 512-dimensional space, applies cross-modal attention for information fusion, and outputs multi-label pathology predictions.
• Figure 3: Parameter-Efficient Methods vs Full Fine-tuning Performance. All PET methods achieve AUROC values exceeding 0.89 using 2.51% trainable parameters, while full fine-tuning achieves 0.770 AUROC using 100% parameters.
• Figure 4: Parameter Efficiency Analysis. Scatter plot shows the relationship between trainable parameters and AUROC performance. PET methods cluster in the high-performance, low-parameter region, while full fine-tuning requires 39.8× more parameters for lower performance.
• Figure 5: Per-Pathology AUROC Performance Analysis across 14 chest pathologies. PET methods consistently achieve performance exceeding 0.85 AUROC across all pathologies, demonstrating robustness across diverse conditions, whereas full fine-tuning exhibits significant variability.