Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PromptGate Client Adaptive Vision Language Gating for Open Set Federated Active Learning

Adea Nesturi, David Dueñas Gaviria, Jiajun Zeng, Shadi Albarqouni

Abstract

Deploying medical AI across resource-constrained institutions demands data-efficient learning pipelines that respect patient privacy. Federated Learning (FL) enables collaborative medical AI without centralising data, yet real-world clinical pools are inherently open-set, containing out-of-distribution (OOD) noise such as imaging artifacts and wrong modalities. Standard Active Learning (AL) query strategies mistake this noise for informative samples, wasting scarce annotation budgets. We propose PromptGate, a dynamic VLM-gated framework for Open-Set Federated AL that purifies unlabeled pools before querying. PromptGate introduces a federated Class-Specific Context Optimization: lightweight, learnable prompt vectors that adapt a frozen BiomedCLIP backbone to local clinical domains and aggregate globally via FedAvg -- without sharing patient data. As new annotations arrive, prompts progressively sharpen the ID/OOD boundary, turning the VLM into a dynamic gatekeeper that is strategy-agnostic: a plug-and-play pre-selection module enhancing any downstream AL strategy. Experiments on distributed dermatology and breast imaging benchmarks show that while static VLM prompting degrades to 50% ID purity, PromptGate maintains $>$95% purity with 98% OOD recall.

PromptGate Client Adaptive Vision Language Gating for Open Set Federated Active Learning

Abstract

Deploying medical AI across resource-constrained institutions demands data-efficient learning pipelines that respect patient privacy. Federated Learning (FL) enables collaborative medical AI without centralising data, yet real-world clinical pools are inherently open-set, containing out-of-distribution (OOD) noise such as imaging artifacts and wrong modalities. Standard Active Learning (AL) query strategies mistake this noise for informative samples, wasting scarce annotation budgets. We propose PromptGate, a dynamic VLM-gated framework for Open-Set Federated AL that purifies unlabeled pools before querying. PromptGate introduces a federated Class-Specific Context Optimization: lightweight, learnable prompt vectors that adapt a frozen BiomedCLIP backbone to local clinical domains and aggregate globally via FedAvg -- without sharing patient data. As new annotations arrive, prompts progressively sharpen the ID/OOD boundary, turning the VLM into a dynamic gatekeeper that is strategy-agnostic: a plug-and-play pre-selection module enhancing any downstream AL strategy. Experiments on distributed dermatology and breast imaging benchmarks show that while static VLM prompting degrades to 50% ID purity, PromptGate maintains 95% purity with 98% OOD recall.
Paper Structure (13 sections, 5 equations, 3 figures, 2 tables)

This paper contains 13 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. VLM with Global and Local Learnable Prompts
  4. VLM-Gated Pseudo-Labeling and Acquisition
  5. Acquisition on the Gated Pool.
  6. Prompt Learning and Federated Aggregation
  7. Experimental Design and Results
  8. Baselines.
  9. Datasets.
  10. Evaluation Metrics.
  11. Implementation details.
  12. PromptGate vs. Static and Coldstart
  13. Discussion and Conclusion

Figures (3)

  • Figure 1: Overview of PromptGate for OS-FAL.
  • Figure 2: Average BMA, QP, and AQR across all OS-FAL methods for different VLM filter variants. Top row: FedISIC. Bottom row: FedEMBED.
  • Figure 3: Average BMA, QP, and AQR across all AL methods for different VLM filter variants in FedISIC