Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Large Language Models for Image Classification of Marine Mammals

Yijiashun Qi, Shuzhang Cai, Zunduo Zhao, Jiaming Li, Yanbin Lin, Zhiqiang Wang

TL;DR

This work builds a benchmark dataset with 1,423 images of 65 kinds of marine mammals, where each animal is uniquely classified into different levels of class, ranging from species-level to medium-level to group-level, and evaluates several approaches for classifying these marine mammals.

Abstract

As Artificial Intelligence (AI) has developed rapidly over the past few decades, the new generation of AI, Large Language Models (LLMs) trained on massive datasets, has achieved ground-breaking performance in many applications. Further progress has been made in multimodal LLMs, with many datasets created to evaluate LLMs with vision abilities. However, none of those datasets focuses solely on marine mammals, which are indispensable for ecological equilibrium. In this work, we build a benchmark dataset with 1,423 images of 65 kinds of marine mammals, where each animal is uniquely classified into different levels of class, ranging from species-level to medium-level to group-level. Moreover, we evaluate several approaches for classifying these marine mammals: (1) machine learning (ML) algorithms using embeddings provided by neural networks, (2) influential pre-trained neural networks, (3) zero-shot models: CLIP and LLMs, and (4) a novel LLM-based multi-agent system (MAS). The results demonstrate the strengths of traditional models and LLMs in different aspects, and the MAS can further improve the classification performance. The dataset is available on GitHub: https://github.com/yeyimilk/LLM-Vision-Marine-Animals.git.

Benchmarking Large Language Models for Image Classification of Marine Mammals

TL;DR

This work builds a benchmark dataset with 1,423 images of 65 kinds of marine mammals, where each animal is uniquely classified into different levels of class, ranging from species-level to medium-level to group-level, and evaluates several approaches for classifying these marine mammals.

Abstract

As Artificial Intelligence (AI) has developed rapidly over the past few decades, the new generation of AI, Large Language Models (LLMs) trained on massive datasets, has achieved ground-breaking performance in many applications. Further progress has been made in multimodal LLMs, with many datasets created to evaluate LLMs with vision abilities. However, none of those datasets focuses solely on marine mammals, which are indispensable for ecological equilibrium. In this work, we build a benchmark dataset with 1,423 images of 65 kinds of marine mammals, where each animal is uniquely classified into different levels of class, ranging from species-level to medium-level to group-level. Moreover, we evaluate several approaches for classifying these marine mammals: (1) machine learning (ML) algorithms using embeddings provided by neural networks, (2) influential pre-trained neural networks, (3) zero-shot models: CLIP and LLMs, and (4) a novel LLM-based multi-agent system (MAS). The results demonstrate the strengths of traditional models and LLMs in different aspects, and the MAS can further improve the classification performance. The dataset is available on GitHub: https://github.com/yeyimilk/LLM-Vision-Marine-Animals.git.

Paper Structure

This paper contains 20 sections, 1 equation, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Research on Marine animals in Aquatic Environments
  4. AI Techniques in Animal Recognition and Data Scarcity for Marine Mammals
  5. LLMs-based Multi-Agent Systems
  6. Dataset
  7. Experimental Settings
  8. Overview of Model Categories
  9. Evaluation Metrics
  10. Machine Learning settings
  11. Deep Learning settings
  12. Multimodal LLMs
  13. Prompting strategies
  14. LLM-based MAS
  15. Analysis and Discussions
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Image samples from the proposed dataset.
  • Figure 2: The process of the LLM-based MAS.
  • Figure 3: The figure shows the hierarchical structure of the labels used in the marine mammal dataset. There are three levels: 4 groups, 10 medium sub-groups, and 65 species, with each medium sub-group represented by 1 species.