Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-modal learning for plankton recognition

Joona Kareinen, Veikka Immonen, Tuomas Eerola, Lumi Haraguchi, Lasse Lensu, Kaisa Kraft, Sanna Suikkanen, Heikki Kälviäinen

Abstract

This paper considers self-supervised cross-modal coordination as a strategy enabling utilization of multiple modalities and large volumes of unlabeled plankton data to build models for plankton recognition. Automated imaging instruments facilitate the continuous collection of plankton image data on a large scale. Current methods for automatic plankton image recognition rely primarily on supervised approaches, which require labeled training sets that are labor-intensive to collect. On the other hand, some modern plankton imaging instruments complement image information with optical measurement data, such as scatter and fluorescence profiles, which currently are not widely utilized in plankton recognition. In this work, we explore the possibility of using such measurement data to guide the learning process without requiring manual labeling. Inspired by the concepts behind Contrastive Language-Image Pre-training, we train encoders for both modalities using only binary supervisory information indicating whether a given image and profile originate from the same particle or from different particles. For plankton recognition, we employ a small labeled gallery of known plankton species combined with a $k$-NN classifier. This approach yields a recognition model that is inherently multimodal, i.e., capable of utilizing information extracted from both image and profile data. We demonstrate that the proposed method achieves high recognition accuracy while requiring only a minimal number of labeled images. Furthermore, we show that the approach outperforms an image-only self-supervised baseline. Code available at https://github.com/Jookare/cross-modal-plankton.

Cross-modal learning for plankton recognition

Abstract

This paper considers self-supervised cross-modal coordination as a strategy enabling utilization of multiple modalities and large volumes of unlabeled plankton data to build models for plankton recognition. Automated imaging instruments facilitate the continuous collection of plankton image data on a large scale. Current methods for automatic plankton image recognition rely primarily on supervised approaches, which require labeled training sets that are labor-intensive to collect. On the other hand, some modern plankton imaging instruments complement image information with optical measurement data, such as scatter and fluorescence profiles, which currently are not widely utilized in plankton recognition. In this work, we explore the possibility of using such measurement data to guide the learning process without requiring manual labeling. Inspired by the concepts behind Contrastive Language-Image Pre-training, we train encoders for both modalities using only binary supervisory information indicating whether a given image and profile originate from the same particle or from different particles. For plankton recognition, we employ a small labeled gallery of known plankton species combined with a -NN classifier. This approach yields a recognition model that is inherently multimodal, i.e., capable of utilizing information extracted from both image and profile data. We demonstrate that the proposed method achieves high recognition accuracy while requiring only a minimal number of labeled images. Furthermore, we show that the approach outperforms an image-only self-supervised baseline. Code available at https://github.com/Jookare/cross-modal-plankton.
Paper Structure (21 sections, 7 equations, 5 figures, 16 tables)

This paper contains 21 sections, 7 equations, 5 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Plankton recognition
  4. Multimodal learning
  5. Methods
  6. Contrastive Image--Profile Pre-training
  7. InfoNCE loss.
  8. Sigmoid loss.
  9. Nearest Neighbor classification
  10. Experiments
  11. Data
  12. Data pre-processing and augmentation
  13. Experimental Setup
  14. Results
  15. In-domain Evaluation.
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Overview of the proposed multimodal plankton recognition model for CytoSense capable of classifying plankton species using (a) only bright-field images, (b) only optical profiles, or (c) both at the same time.
  • Figure 2: Overview of the proposed multimodal recognition framework. (a) During pre-training, paired image--profile samples are aligned in a shared embedding space. (b) At inference, classification is performed using nearest-neighbor search in the embedding space.
  • Figure 3: Example image and profile data
  • Figure 4: Pre-processing and data augmentations: (a) original profile and image, (b) same pair pre-processed
  • Figure 5: Average accuracy and standard deviation as a function of gallery set size, with $n$ being the number of multimodal samples in the gallery set (I+P $\rightarrow$ I+P)