A new dataset for measuring the performance of blood vessel segmentation methods under distribution shifts

TL;DR

VessMAP is introduced, an annotated and highly heterogeneous blood vessel segmentation dataset acquired by carefully sampling relevant images from a large non-annotated dataset containing fluorescence microscopy images that can be used for measuring the performance of segmentation algorithms on samples that are highly distinct from each other.

Abstract

Creating a dataset for training supervised machine learning algorithms can be a demanding task. This is especially true for medical image segmentation since one or more specialists are usually required for image annotation, and creating ground truth labels for just a single image can take up to several hours. In addition, it is paramount that the annotated samples represent well the different conditions that might affect the imaged tissues as well as possible changes in the image acquisition process. This can only be achieved by considering samples that are typical in the dataset as well as atypical, or even outlier, samples. We introduce VessMAP, a heterogeneous blood vessel segmentation dataset acquired by carefully sampling relevant images from a larger non-annotated dataset. A methodology was developed to select both prototypical and atypical samples from the base dataset, thus defining an assorted set of images that can be used for measuring the performance of segmentation algorithms on samples that are highly distinct from each other. To demonstrate the potential of the new dataset, we show that the validation performance of a neural network changes significantly depending on the splits used for training the network.

Figures (7)

• Figure 1: An example of seven regions extracted from a single sample. The four corners along with the central region can capture most of the illumination inhomogeneities that may occur due to uneven illumination of the samples. Besides these five regions, two additional random regions are also drawn for each image.
• Figure 2: Representation of the mapping procedure applied to a set $D$ of samples, followed by the feature space discretization. Here, we consider $D$ as a set of blood vessel images. In this example, each image of $D$ is mapped to a 2-d position in the new feature space (b). In (c), the mapped points (light-red points) are moved to a new position (red points) within a regular grid defined by Equation \ref{['eq:resample']}.
• Figure 3: Illustration of the proposed sampling protocol. $k$ random points (green dots) are drawn from the sampling set $D_{sset}$ (blue dots). The subset of sampled data points is defined by the data points that are closest to each drawn point (orange stars). Red squares represent the remaining data points that were not selected.
• Figure 4: Histograms of the four features calculated from the dataset. Blue bars correspond to the distribution of the full data. Orange bars correspond to the distribution of the sampled subset. Note that the frequencies were normalized by their sum.
• Figure 5: PCA of the blood vessel dataset. The original features with z-score normalization was used. Red points correspond to the sampled subset obtained by the sampling methodology. Other points correspond to the unselected points, with their colors representing the value of each one of the four original metrics: vessel density, contrast, medial line heterogeneity, and image noise.