Position: Restructuring of Categories and Implementation of Guidelines Essential for VLM Adoption in Healthcare

TL;DR

This position paper addresses the lack of standardized reporting for vision-language models (VLMs) in healthcare, which hampers reproducibility and clinical adoption. It introduces a four-category framework (pretraining, domain-specific finetuning, task-specific finetuning, prompting) with category-specific reporting standards and an appendix checklist to guide publication and evaluation. The authors detail requirements for model design, training objectives, datasets, performance metrics, and multi-phase bias analysis, aiming to enhance transparency, comparability, and trust in VLM-based healthcare applications. By aligning reporting with the VLM study category and advocating for rigorous, multi-phase evaluation, the work seeks to accelerate safe, effective deployment of VLMs in clinical settings.

Abstract

The intricate and multifaceted nature of vision language model (VLM) development, adaptation, and application necessitates the establishment of clear and standardized reporting protocols, particularly within the high-stakes context of healthcare. Defining these reporting standards is inherently challenging due to the diverse nature of studies involving VLMs, which vary significantly from the development of all new VLMs or finetuning for domain alignment to off-the-shelf use of VLM for targeted diagnosis and prediction tasks. In this position paper, we argue that traditional machine learning reporting standards and evaluation guidelines must be restructured to accommodate multiphase VLM studies; it also has to be organized for intuitive understanding of developers while maintaining rigorous standards for reproducibility. To facilitate community adoption, we propose a categorization framework for VLM studies and outline corresponding reporting standards that comprehensively address performance evaluation, data reporting protocols, and recommendations for manuscript composition. These guidelines are organized according to the proposed categorization scheme. Lastly, we present a checklist that consolidates reporting standards, offering a standardized tool to ensure consistency and quality in the publication of VLM-related research.

Figures (3)

• Figure 1: Conceptual framework of the proposed VLM study categorization where the arrows represent unidirectional inter-dependencies between the VLM categories.
• Figure 2: Conceptual categorization of VLM datasets based on our proposed study categorization.
• Figure 3: Conceptual diagram for VLM performance reporting.