Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-supervised Visualisation of Medical Image Datasets

Ifeoma Veronica Nwabufo, Jan Niklas Böhm, Philipp Berens, Dmitry Kobak

TL;DR

The paper addresses the challenge of visualising high-dimensional self-supervised medical-image representations by applying $t$-SimCNE to a diverse set of dermatology, histology, and blood-microscopy datasets. It demonstrates that domain-tailored rotational augmentations improve 2D cluster separability and that the resulting embeddings reveal medically meaningful structure and artefacts, often outperforming $t$-SNE on pretrained features. The proposed approach yields actionable 2D maps for data exploration and annotation, with the added benefit of enabling out-of-sample embedding due to its parametric nature. Overall, the work provides a practical, interpretable tool for visual analytics in medical imaging using self-supervised contrastive representations.

Abstract

Self-supervised learning methods based on data augmentations, such as SimCLR, BYOL, or DINO, allow obtaining semantically meaningful representations of image datasets and are widely used prior to supervised fine-tuning. A recent self-supervised learning method, $t$-SimCNE, uses contrastive learning to directly train a 2D representation suitable for visualisation. When applied to natural image datasets, $t$-SimCNE yields 2D visualisations with semantically meaningful clusters. In this work, we used $t$-SimCNE to visualise medical image datasets, including examples from dermatology, histology, and blood microscopy. We found that increasing the set of data augmentations to include arbitrary rotations improved the results in terms of class separability, compared to data augmentations used for natural images. Our 2D representations show medically relevant structures and can be used to aid data exploration and annotation, improving on common approaches for data visualisation.

Self-supervised Visualisation of Medical Image Datasets

TL;DR

The paper addresses the challenge of visualising high-dimensional self-supervised medical-image representations by applying -SimCNE to a diverse set of dermatology, histology, and blood-microscopy datasets. It demonstrates that domain-tailored rotational augmentations improve 2D cluster separability and that the resulting embeddings reveal medically meaningful structure and artefacts, often outperforming -SNE on pretrained features. The proposed approach yields actionable 2D maps for data exploration and annotation, with the added benefit of enabling out-of-sample embedding due to its parametric nature. Overall, the work provides a practical, interpretable tool for visual analytics in medical imaging using self-supervised contrastive representations.

Abstract

Self-supervised learning methods based on data augmentations, such as SimCLR, BYOL, or DINO, allow obtaining semantically meaningful representations of image datasets and are widely used prior to supervised fine-tuning. A recent self-supervised learning method, -SimCNE, uses contrastive learning to directly train a 2D representation suitable for visualisation. When applied to natural image datasets, -SimCNE yields 2D visualisations with semantically meaningful clusters. In this work, we used -SimCNE to visualise medical image datasets, including examples from dermatology, histology, and blood microscopy. We found that increasing the set of data augmentations to include arbitrary rotations improved the results in terms of class separability, compared to data augmentations used for natural images. Our 2D representations show medically relevant structures and can be used to aid data exploration and annotation, improving on common approaches for data visualisation.
Paper Structure (11 sections, 2 equations, 5 figures, 4 tables)

This paper contains 11 sections, 2 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Background: SimCLR and $t$-SimCNE
  4. Experimental setup
  5. Datasets
  6. Augmentations
  7. Architecture and training
  8. Baselines
  9. Evaluation
  10. Results
  11. Discussion

Figures (5)

  • Figure 1: (a) In $t$-SimCNE, the network is trained to map two random augmentations of an input image to close locations in the 2D output space. (b) Augmentations used for natural images in $t$-SimCNE. (c) Additional augmentations suggested here for medical images.
  • Figure 2: Visualisations of the Leukemia dataset. Small classes shown in black ('OTH' in the legend). $k$NN accuracy and silhouette scores shown in each panel. (a)$t$-SNE of the original images in the pixel space. (b)$t$-SNE of the 512-dimensional representation obtained via an ImageNet-pretrained ResNet18 network. (c)$t$-SimCNE using the same augmentations as in boehm2023unsupervised. (d)$t$-SimCNE using augmentations including 90° rotations and flips. Note that the EBO class is well separated here, despite only consisting of 78 images.
  • Figure 3: (a)$t$-SimCNE visualisation of the Leukemia dataset. Only a subset of classes is listed in the legend. (b)$t$-SimCNE visualisation of the Blood mnist dataset. (c)$t$-SimCNE visualisation of the Derma mnist dataset. In all three cases, we used augmentations including 90° rotations and vertical flips.
  • Figure 4: $t$-SimCNE visualisation of the Path mnist dataset. Colours correspond to classes. Images correspond to three random points close to the tip of the annotation line.
  • Figure 5: (a)$t$-SimCNE visualisation of the PCam16 dataset. (b) We superimposed a $10\times 10$ grid over the embedding and selected one image in each square. Frame colours show image classes. If a square had fewer than 100 images, no image was shown.