Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis

Prateek Verma, Minh-Hao Van, Xintao Wu

TL;DR

The study evaluates large vision-language models (ChatGPT-4, Gemini Pro-Vision) and the segmentation model SAM on microscopy images from SEM and BBBC005 datasets across four tasks: classification, segmentation, counting, and VQA. It demonstrates that ChatGPT and Gemini can interpret visual features in microscopy data and perform reasonable classification, while SAM shows strong segmentation capabilities but struggles with artefact overlap; LLaVA generally underperforms in this domain. Performance is highly task- and dataset-dependent, with domain-adaptive prompting and parameter tuning (e.g., SAM custom settings) yielding notable gains in counting and segmentation, but impurities and complex textures still hinder accuracy compared to domain experts. Overall, the results highlight both the promise and current limitations of foundation models in specialized scientific image analysis, pointing to the need for targeted fine-tuning and domain-specific evaluation protocols. The work provides a framework for zero-shot and minimally supervised assessment of multimodal models in microscopy, with potential impact on automated analysis workflows in biology, medicine, and materials science.

Abstract

Vision language models (VLMs) have recently emerged and gained the spotlight for their ability to comprehend the dual modality of image and textual data. VLMs such as LLaVA, ChatGPT-4, and Gemini have recently shown impressive performance on tasks such as natural image captioning, visual question answering (VQA), and spatial reasoning. Additionally, a universal segmentation model by Meta AI, Segment Anything Model (SAM) shows unprecedented performance at isolating objects from unforeseen images. Since medical experts, biologists, and materials scientists routinely examine microscopy or medical images in conjunction with textual information in the form of captions, literature, or reports, and draw conclusions of great importance and merit, it is indubitably essential to test the performance of VLMs and foundation models such as SAM, on these images. In this study, we charge ChatGPT, LLaVA, Gemini, and SAM with classification, segmentation, counting, and VQA tasks on a variety of microscopy images. We observe that ChatGPT and Gemini are impressively able to comprehend the visual features in microscopy images, while SAM is quite capable at isolating artefacts in a general sense. However, the performance is not close to that of a domain expert - the models are readily encumbered by the introduction of impurities, defects, artefact overlaps and diversity present in the images.

Beyond Human Vision: The Role of Large Vision Language Models in Microscope Image Analysis

TL;DR

The study evaluates large vision-language models (ChatGPT-4, Gemini Pro-Vision) and the segmentation model SAM on microscopy images from SEM and BBBC005 datasets across four tasks: classification, segmentation, counting, and VQA. It demonstrates that ChatGPT and Gemini can interpret visual features in microscopy data and perform reasonable classification, while SAM shows strong segmentation capabilities but struggles with artefact overlap; LLaVA generally underperforms in this domain. Performance is highly task- and dataset-dependent, with domain-adaptive prompting and parameter tuning (e.g., SAM custom settings) yielding notable gains in counting and segmentation, but impurities and complex textures still hinder accuracy compared to domain experts. Overall, the results highlight both the promise and current limitations of foundation models in specialized scientific image analysis, pointing to the need for targeted fine-tuning and domain-specific evaluation protocols. The work provides a framework for zero-shot and minimally supervised assessment of multimodal models in microscopy, with potential impact on automated analysis workflows in biology, medicine, and materials science.

Abstract

Vision language models (VLMs) have recently emerged and gained the spotlight for their ability to comprehend the dual modality of image and textual data. VLMs such as LLaVA, ChatGPT-4, and Gemini have recently shown impressive performance on tasks such as natural image captioning, visual question answering (VQA), and spatial reasoning. Additionally, a universal segmentation model by Meta AI, Segment Anything Model (SAM) shows unprecedented performance at isolating objects from unforeseen images. Since medical experts, biologists, and materials scientists routinely examine microscopy or medical images in conjunction with textual information in the form of captions, literature, or reports, and draw conclusions of great importance and merit, it is indubitably essential to test the performance of VLMs and foundation models such as SAM, on these images. In this study, we charge ChatGPT, LLaVA, Gemini, and SAM with classification, segmentation, counting, and VQA tasks on a variety of microscopy images. We observe that ChatGPT and Gemini are impressively able to comprehend the visual features in microscopy images, while SAM is quite capable at isolating artefacts in a general sense. However, the performance is not close to that of a domain expert - the models are readily encumbered by the introduction of impurities, defects, artefact overlaps and diversity present in the images.
Paper Structure (15 sections, 13 figures, 2 tables)

This paper contains 15 sections, 13 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Datasets
  4. Vision Language Models Used
  5. Tasks
  6. Classification
  7. Segmentation
  8. Counting
  9. Visual Question Answering (VQA)
  10. Results and Discussion
  11. Classification
  12. Segmentation
  13. Counting
  14. Visual Question Answering (VQA)
  15. Conclusions

Figures (13)

  • Figure 1: An illustration of the tasks, models, and datasets utilized in this study along with the relational mapping provided as a visual aid to the reader; the details for each box and mapping will become more apparent with their progression in the manuscript.
  • Figure 2: Sample images from each dataset. The classes for the NFFA SEM dataset are defined in Table \ref{['tab:datasets_1']}. For the BBBC005 dataset, one image each for a low cell count and a high cell count are shown (top row) along with the corresponding segmentation ground truth images below (bottom row). The images on the left belong to the w1 subset, while those on the right belong to the w2 subset.
  • Figure 3: ChatGPT considered the scale bar while predicting the class of this image as fibers correctly. It is possible that this judgement helped ChatGPT to distinguish it from nanowires. Correct key responses are colored in green.
  • Figure 4: Confusion matrices for the classification of ten categories in the NFFA dataset by (a) LLaVA on the entire dataset (numbers denote percentages with respect to the number of actual samples present in the respective classes), (b) LLaVA on 25 images per class of the NFFA-RS, (c) Gemini on the entire dataset (numbers denote percentages as in (a)), (d) Gemini on 25 images per class of the NFFA-RS, and (e) ChatGPT on 25 images per class of the NFFA-RS. Class names have been abbreviated as noted in Table \ref{['tab:datasets_1']}.
  • Figure 5: Dice scores, as calculated for segmentation by SAM-standard and SAM-custom of the 600 w1 (top row) and 600 w2 (bottom row) images of the BBBC005 dataset, plotted against cell counts.
  • ...and 8 more figures