Enhancing Cell Tracking with a Time-Symmetric Deep Learning Approach

TL;DR

This work tackles the challenge of accurate live-cell tracking in video microscopy by moving beyond frame-to-frame tracking and introducing a time-symmetric, multi-frame approach that leverages the local spatio-temporal neighborhood. The pipeline combines a Mask R-CNN-based instance segmentation stage (via Detectron2) with a time-symmetric local tracker (built on DeepLabV3+) and a global Hungarian-assignment step using an IOU-based similarity, enabling frame skipping up to $2TR$ while maintaining coherent tracks. Evaluations on budding yeast data and multiple synthetic datasets show state-of-the-art-like performance, with robustness to artifacts and reduced sensitivity to segmentation quality through joint segmentation-tracking dynamics and interpolation. The method generalizes to diverse cell types given ample labeled data and is made available in an open-source repository, offering a scalable, data-driven solution for cellular tracking in biological research.

Abstract

The accurate tracking of live cells using video microscopy recordings remains a challenging task for popular state-of-the-art image processing based object tracking methods. In recent years, several existing and new applications have attempted to integrate deep-learning based frameworks for this task, but most of them still heavily rely on consecutive frame based tracking embedded in their architecture or other premises that hinder generalized learning. To address this issue, we aimed to develop a new deep-learning based tracking method that relies solely on the assumption that cells can be tracked based on their spatio-temporal neighborhood, without restricting it to consecutive frames. The proposed method has the additional benefit that the motion patterns of the cells can be learned completely by the predictor without any prior assumptions, and it has the potential to handle a large number of video frames with heavy artifacts. The efficacy of the proposed method is demonstrated through biologically motivated validation strategies and compared against multiple state-of-the-art cell tracking methods.

Figures (12)

• Figure 1: A showcase of robust segmentation results, depicting accurate cell detection and segmentation under varying lighting conditions, while effectively avoiding detection of obviously dead cells.
• Figure 2: Depiction of input and output information structure of local tracking via segmentation with local tracking range ($TR$) = 2. Notice that while the left to right positive temporal direction of the data is recognizable from cell growth, and the model learns this during training, the architecture itself has no built in directional preference, resulting in a completely data-driven estimation for local tracking.
• Figure 3: A side by side display of tracking results, demonstrating successful tracking of all cells in the same recording with a temporal difference of 8 frames. New cells were appropriately assigned new IDs while maintaining consistent tracking of existing cells.
• Figure 4: Schematic structure of the metric similarity measurement step between $2TR+1$ long local tracks of different cell instances on frames with a temporal distance of $\Delta t$. The solid lines indicate the central segmentation instances with a $\Delta t$ temporal distance to be matched, while the arrows indicate the similarity metric between the segmentation estimates for each time frame. Subsequently, the individual metric results are averaged to obtain a single measure of similarity.
• Figure 5: Illustration of the metric similarity based assignment step using the Hungarian method for non-square matrices. Newly unassigned cells, such as cell number 4 receive a new ID, while the previously assigned cells retain the ID of their corresponding previous instances.