Self-supervised Representation Learning for Cell Event Recognition through Time Arrow Prediction

TL;DR

This work demonstrates that using feature maps obtained from self-supervised representation learning on time arrow prediction for the downstream supervised task of cell event recognition can achieve better performance with limited annotation compared to models trained from end to end using fully supervised approach.

Abstract

The spatio-temporal nature of live-cell microscopy data poses challenges in the analysis of cell states which is fundamental in bioimaging. Deep-learning based segmentation or tracking methods rely on large amount of high quality annotations to work effectively. In this work, we explore an alternative solution: using feature maps obtained from self-supervised representation learning (SSRL) on time arrow prediction (TAP) for the downstream supervised task of cell event recognition. We demonstrate through extensive experiments and analysis that this approach can achieve better performance with limited annotation compared to models trained from end to end using fully supervised approach. Our analysis also provides insight into applications of the SSRL using TAP in live-cell microscopy.

Figures (7)

• Figure 1: Illustration of our data pre-processing. Starting with images and masks that are indexed by time, we take crops and get (crops, masks) pairs where crops are taken at the same location of the images at consecutive time points. We then apply our labelling criteria to turn the masks into binary labels.
• Figure 2: Example images overlaid with attribution map from Grad-CAM SelvarajuGrad-CAM2017. Figure \ref{['fig:grad cam training examples']} shows one image from the training set and Figure \ref{['fig:grad cam validation examples']} shows one from the validation set which is taken from the same experiment but a different location of the live cell sample. The numbered crops are showing the top $8$ regions ranked by the Grad-CAM scores, so the higher the more influential the pixels in the region have on the time arrow prediction. The bottom rows show the view at current time point and one frame afterwards. We can see many of these highlighted regions contain changes in cell morphology resulted from divisions.
• Figure 3: Distribution of false positive predictions by time. The percentage at a specific time index is calculated by dividing the number of false positive predictions at that time point by the total number of false positives made by the model over the entire test dataset. We also plot the true positives and ground truth positives for comparison.
• Figure 4: Distribution of false negative predictions by time. The percentage at a specific time index is calculated similarly over the negative datapoints. We also plot the true negatives and ground truth negatives for comparison.
• Figure 5: Reliability diagrams of the original and calibrated models. The red dashed line indicates perfect calibration where the accuracy (for class 1) is equal to the confidence. At a fixed confidence level, the prediction is over-confident if the accuracy is below the dashed line. In the left column, \ref{['fig:calibration linear original']} and \ref{['fig:calibration resnet original']} show the reliability diagrams for the model with linear and ResNet head before calibration using temperature scaling, and the right column show those after calibration. The expected calibration error (ECE) is shown in the top left corner.