Cell Behavior Video Classification Challenge, a benchmark for computer vision methods in time-lapse microscopy

TL;DR

The Cell Behavior Video Classification Challenge is organized, benchmarking 35 methods based on three approaches: classification of tracking-derived features, end-to-end deep learning architectures to directly learn spatiotemporal features from the entire video sequence without explicit cell tracking, or ensembling tracking-derived with image-derived features.

Abstract

The classification of microscopy videos capturing complex cellular behaviors is crucial for understanding and quantifying the dynamics of biological processes over time. However, it remains a frontier in computer vision, requiring approaches that effectively model the shape and motion of objects without rigid boundaries, extract hierarchical spatiotemporal features from entire image sequences rather than static frames, and account for multiple objects within the field of view. To this end, we organized the Cell Behavior Video Classification Challenge (CBVCC), benchmarking 35 methods based on three approaches: classification of tracking-derived features, end-to-end deep learning architectures to directly learn spatiotemporal features from the entire video sequence without explicit cell tracking, or ensembling tracking-derived with image-derived features. We discuss the results achieved by the participants and compare the potential and limitations of each approach, serving as a basis to foster the development of computer vision methods for studying cellular dynamics.

Figures (3)

• Figure 1: IVM data acquisition, annotation, and dataset composition. A. Simplified representation of the IVM video acquisition. The flank skin of anesthetized and immobilized mice is surgically exposed and stabilized using a specialized imaging window. Imaging is performed using a multiphoton microscope with point-wise excitation. B. Example of acquired IVM video, consisting of a z-stack of parallel image planes over time across multiple fluorescence channels. C. Cropped sequence of frames highlighting the dynamic movement of a migrating T cell. D. Schematic representation of the annotation process: an operator semi-manually navigated through the video frames to select coordinates for video-patch extraction. Class 1 video-patches were annotated to include instances where the central cell in the middle frame changes its direction of movement. Class 0 video-patches were annotated to include cells that move in a linear trajectory, stationary cells, or some background regions with no visible cells. E. Distribution of the number of video-patches annotated as Class 0 and Class 1, divided among the three datasets. F. Distribution of the number of cells and the signal-to-noise ratio (SNR) per video-patch across the three datasets.
• Figure 2: Approaches for the classification of CBVCC video-patches.A. Track-based methods involve segmentation, tracking, feature extraction using either handcrafted features or deep neural networks (DNNs), and classification through machine learning (ML) models or a multilayer perceptron (MLP). B. End-to-end DL methods include an automatic feature extraction module that processes video-patches using a DNN, followed by an MLP for class prediction.
• Figure 3: Performance Evaluation.A. ROC curve of the participating methods for the video-patch classification task on the validation set. The curves represent the performance of only the best model from each team, selected from their multiple submissions. B. ROC curve of the participating methods on the test set, illustrating the trade-off between sensitivity and specificity and highlighting performance differences with the validation set. C. Performance on the test set as a function of the number of cells in the video-patch. The bar plot below ranks each team based on the number of cells. D. Performance trend on the test set as a function of the signal-to-noise ratio (SNR) of the video-patch. A maximum SNR of 12.0 was used to maintain an adequate number of video-patches. The accompanying bar plot ranks each team based on the SNR. E. Baseline logistic regression performance in terms of balanced accuracy. F. Baseline logistic regression performance in terms of overall score. In E and F gray dots represent results from 5 repeats of 5-fold cross-validation (25 models), with average performance (|) shown on the training (left) and validation (right) folds. Colored dots represent results from training on the entire training dataset and evaluation on the CBVCC training set ($\blacktriangle$), validation set ($\bullet$), and test set ($\blacksquare$). “basic” indicates the set of standard motility features, “all” the extended set including the two additional hand-crafted features, auto automated tracks, and manual manually annotated tracks.