Patient-Conditioned Adaptive Offsets for Reliable Diagnosis across Subgroups

TL;DR

Medical diagnoses often suffer from performance gaps across patient subgroups due to heterogeneous disease prevalence and presentation. HyperAdapt encodes patient attributes into a compact embedding that conditions small residual parameter offsets generated by a hyper-adapter, which modulates a frozen backbone via low-rank adapters and shared-generation blocks. Across Fitzpatrick-17k, ODIR-5k, and PAD-UFES-20 with backbones such as ResNet-50 and Swin-T, HyperAdapt yields consistent subgroup improvements while preserving overall accuracy, including a 4.1% recall and 4.4% F1 boost on PAD-UFES-20 relative to strong baselines. This context-conditioned approach mirrors clinical reasoning, providing scalable, efficient, and interpretable improvements in subgroup reliability for medical imaging diagnosis.

Abstract

AI models for medical diagnosis often exhibit uneven performance across patient populations due to heterogeneity in disease prevalence, imaging appearance, and clinical risk profiles. Existing algorithmic fairness approaches typically seek to reduce such disparities by suppressing sensitive attributes. However, in medical settings these attributes often carry essential diagnostic information, and removing them can degrade accuracy and reliability, particularly in high-stakes applications. In contrast, clinical decision making explicitly incorporates patient context when interpreting diagnostic evidence, suggesting a different design direction for subgroup-aware models. In this paper, we introduce HyperAdapt, a patient-conditioned adaptation framework that improves subgroup reliability while maintaining a shared diagnostic model. Clinically relevant attributes such as age and sex are encoded into a compact embedding and used to condition a hypernetwork-style module, which generates small residual modulation parameters for selected layers of a shared backbone. This design preserves the general medical knowledge learned by the backbone while enabling targeted adjustments that reflect patient-specific variability. To ensure efficiency and robustness, adaptations are constrained through low-rank and bottlenecked parameterizations, limiting both model complexity and computational overhead. Experiments across multiple public medical imaging benchmarks demonstrate that the proposed approach consistently improves subgroup-level performance without sacrificing overall accuracy. On the PAD-UFES-20 dataset, our method outperforms the strongest competing baseline by 4.1% in recall and 4.4% in F1 score, with larger gains observed for underrepresented patient populations.

Figures (6)

• Figure 1: Comparison of paradigms for integrating patient attributes. (1) Standard training leads to biased performance; (2) Group-blind suppression degrades overall accuracy; (3) Our HyperAdapt enables context-aware reliability.
• Figure 2: The HyperAdapt framework. Patient attributes are embedded and fused into a patient embedding, which is fed into hyper-adapter layers that generate subgroup-conditioned parameter offsets for selected convolutional and linear layers. These offsets modify the frozen base-model weights to produce adapted parameters used for inference. The adapted model then processes the input image to produce the diagnostic output.
• Figure 3: Group-wise performance comparison across Fitzpatrick-17k, ODIR-5k, and PAD-UFES-20. Each panel shows the change in accuracy, F1, and fairness metrics (Eopp0, Eopp1, Eodds) for each method relative to the vanilla baseline. Green bars denote performance improvements, and red bars denote declines.
• Figure 4: t-SNE Visualization of Contextual Feature Space Organization. The plots compare the feature space (from a single disease class) learned by different methods. (Top Row) On Fitzpatrick17k, features are colored by the 6 ordinal skin types. (Bottom Row) On ORID5k, features are colored by composite gender and age groups.
• Figure 5: Linear probing visualizations of selected metadata attributes on PAD-UFES-20. These visualizations illustrate how clinically relevant metadata manifests structured alignment within the learned image embedding space.