Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

At First Sight: Zero-Shot Classification of Astronomical Images with Large Multimodal Models

Dimitrios Tanoglidis, Bhuvnesh Jain

TL;DR

This work investigates two models, GPT-4o and LLaVA-NeXT, for zero-shot classification of low-surface brightness galaxies and artifacts, as well as morphological classification of galaxies, and shows that with natural language prompts these models achieved significant accuracy without additional training/fine tuning.

Abstract

Vision-Language multimodal Models (VLMs) offer the possibility for zero-shot classification in astronomy: i.e. classification via natural language prompts, with no training. We investigate two models, GPT-4o and LLaVA-NeXT, for zero-shot classification of low-surface brightness galaxies and artifacts, as well as morphological classification of galaxies. We show that with natural language prompts these models achieved significant accuracy (above 80 percent typically) without additional training/fine tuning. We discuss areas that require improvement, especially for LLaVA-NeXT, which is an open source model. Our findings aim to motivate the astronomical community to consider VLMs as a powerful tool for both research and pedagogy, with the prospect that future custom-built or fine-tuned models could perform better.

At First Sight: Zero-Shot Classification of Astronomical Images with Large Multimodal Models

TL;DR

This work investigates two models, GPT-4o and LLaVA-NeXT, for zero-shot classification of low-surface brightness galaxies and artifacts, as well as morphological classification of galaxies, and shows that with natural language prompts these models achieved significant accuracy without additional training/fine tuning.

Abstract

Vision-Language multimodal Models (VLMs) offer the possibility for zero-shot classification in astronomy: i.e. classification via natural language prompts, with no training. We investigate two models, GPT-4o and LLaVA-NeXT, for zero-shot classification of low-surface brightness galaxies and artifacts, as well as morphological classification of galaxies. We show that with natural language prompts these models achieved significant accuracy (above 80 percent typically) without additional training/fine tuning. We discuss areas that require improvement, especially for LLaVA-NeXT, which is an open source model. Our findings aim to motivate the astronomical community to consider VLMs as a powerful tool for both research and pedagogy, with the prospect that future custom-built or fine-tuned models could perform better.
Paper Structure (3 sections, 3 figures)

This paper contains 3 sections, 3 figures.

Table of Contents

  1. Introduction
  2. METHODS
  3. RESULTS AND DISCUSSION

Figures (3)

  • Figure 1: The prompts used to instruct the multimodal models on the two classification tasks (upper part), as well as the resulting confusion matrices, obtained using the GPT-4o (middle part) and LLaVa-NeXT (lower part) models. The confusion matrices compare the predicted vs. the true labels for the example images in our test datasets. The numbers in parentheses correspond to the fraction of images with true label in class $x$ that were predicted to belong to class $y$.
  • Figure 2: Examples of LSBG and artifact images.
  • Figure 3: Examples of images from the four morphological categories in the GalaxyMNIST dataset. Notice the visual similarity between the smooth cigar-shaped and the edge-on disk galaxies.