Evaluating ChatGPT's Performance in Classifying Pneumonia from Chest X-Ray Images

TL;DR

This paper investigates zero-shot classification of pneumonia from chest X-ray images using the multimodal gpt-4o model, guided by four prompt designs that range from minimal output to reasoning-based prompts. On a test set derived from a public chest X-ray pneumonia dataset, the study finds that concise, feature-focused prompts yield the highest accuracy at 74%, while prompts that require longer reasoning provide no performance benefit. The results suggest emerging multimodal capabilities in ChatGPT for medical imaging but also underline substantial limitations in diagnostic reliability without further visual reasoning enhancements and domain-specific fine-tuning. The work highlights prompt design as a critical factor in zero-shot medical image classification and points to the need for cautious clinical deployment and continued research.

Abstract

In this study, we evaluate the ability of OpenAI's gpt-4o model to classify chest X-ray images as either NORMAL or PNEUMONIA in a zero-shot setting, without any prior fine-tuning. A balanced test set of 400 images (200 from each class) was used to assess performance across four distinct prompt designs, ranging from minimal instructions to detailed, reasoning-based prompts. The results indicate that concise, feature-focused prompts achieved the highest classification accuracy of 74\%, whereas reasoning-oriented prompts resulted in lower performance. These findings highlight that while ChatGPT exhibits emerging potential for medical image interpretation, its diagnostic reliability remains limited. Continued advances in visual reasoning and domain-specific adaptation are required before such models can be safely applied in clinical practice.

Figures (1)

• Figure 1: Example chest X-ray images from the test set in the Chest X-Ray Images (Pneumonia) dataset mahmoud2018chestxray. (a) Pneumonia X-ray (b) Normal X-ray.