EmbedTrack -- Simultaneous Cell Segmentation and Tracking Through Learning Offsets and Clustering Bandwidths

TL;DR

EmbedTrack is presented, a single convolutional neural network for simultaneous cell segmentation and tracking which predicts human comprehensible embeddings which performs well on nine 2D data sets from the Cell Tracking Challenge.

Abstract

A systematic analysis of the cell behavior requires automated approaches for cell segmentation and tracking. While deep learning has been successfully applied for the task of cell segmentation, there are few approaches for simultaneous cell segmentation and tracking using deep learning. Here, we present EmbedTrack, a single convolutional neural network for simultaneous cell segmentation and tracking which predicts easy to interpret embeddings. As embeddings, offsets of cell pixels to their cell center and bandwidths are learned. We benchmark our approach on nine 2D data sets from the Cell Tracking Challenge, where our approach performs on seven out of nine data sets within the top 3 contestants including three top 1 performances. The source code is publicly available at https://git.scc.kit.edu/kit-loe-ge/embedtrack.

Figures (4)

• Figure 1: Visualization of the clustering using different clustering bandwidths. The plots show the distance $d(\mathbf{c}, \mathbf{p})$ using Gaussian kernels with different bandwidths $s=s_x=s_y$ in range $[0, 1]$ (heat maps from blue to yellow) of each pixel $\mathbf{p}$ in the plot to the object center $\mathbf{c}$ (cyan) of the ellipsoid object. The red circles are contours where the distance score between the center $\mathbf{c}$ and any pixel $\mathbf{p}$ is $d(\mathbf{c}, \mathbf{p})=0.5$. The points (black) visualize pixels shifted by their predicted offset to the center of the ellipsoid object. To cluster the pixels into instances, points (black) will be assigned to the same instance if they lay within the red circle.
• Figure 2: Method Overview. The approach consists of three steps: (1) prediction of offsets, clustering bandwidths and seediness maps using a CNN, (2) processing the predicted segmentation offsets and clustering bandwidths in a clustering step to retrieve an instance segmentation, and (3) linking the instance segmentation masks over time by processing the by the CNN predicted tracking offsets. (1) The CNN, a branched ERFNet, receives pairs of raw image crops, of time points $t$ and $t-1$. The CNN predicts two sets of segmentation predictions, one set for each of the two time points, and one tracking offset tensor. The offsets and the clustering bandwidths are tensors, where their values along the $x$-and $y$-dimension are shown as maps, where $H$ and $W$ are the height and the width of the raw image crops. (2) The segmentation predictions are processed in a clustering step to retrieve instance segmentation masks for time points $t$ and $t-1$, where gray and black show the prediction of the network for cell and background and the yellow pixels show the cell pixels after adding the predicted offset. The red circles indicate $d^s_{j,i}=0.5$ with the predicted clustering bandwidths. (3) The instance segmentation masks are linked backwards in time using the predicted tracking offsets, which is a tensor here shown with its values along $x$- and $y$-dimension. Raw image crops of the data set BF-C2DL-HSC from the CTC maska_benchmark_2014ulman_objective_2017.
• Figure 3: Clustering step. (A) the seediness map $\mathbf{D}$ is thresholded resulting in a foreground (gray) background (black) prediction. Next, the pixels predicted as foreground are shifted by the predicted segmentation offsets $\mathbf{O}^S$, where the shifted pixels are shown in yellow (B). (C) The shifted pixels are assigned to clusters by selecting an unclustered, shifted pixel (cyan) and calculating its distance $d^S_{j,i}(\mathbf{e}^S_j, \mathbf{e}^S_i)$ to all other pixels $i$-- distance map shown as a heat map. (D) The distance map is thresholded and pixels with a distance score higher than $0.5$ (red circle) are assigned to the same cluster. The steps (C) and (D) are repeated until all pixels are clustered (E). Finally, the clustered pixels are converted into a instance masks, where each cluster represents an instance. Raw image crop of the data set BF-C2DL-HSC from the CTC maska_benchmark_2014ulman_objective_2017.
• Figure 4: Benchmark Data from the CTC. All selected data sets from the CTC on which the EmbedTrack approach was evaluated and trained on. The contrast of the raw images has been adapted using min-max scaling to the percentiles 1 and 99 for visualization purposes. All data sets can be retrieved from http://celltrackingchallenge.net/.