CHOTA: A Higher Order Accuracy Metric for Cell Tracking

TL;DR

The paper tackles the imbalance in cell tracking evaluation where local detection and frame-to-frame associations dominate metrics, often neglecting global lineage and coherence. It introduces CHOTA, the Cell-specific Higher Order Tracking Accuracy, by redefining trajectories to be lineage-aware and extending the HOTA framework with a lineage-based association score A^σ(c). CHOTA aggregates local correctness, global coherence, and lineage consistency through the TPA^σ/ FNA^σ / FPA^σ formulation, and demonstrates via real data and synthetic error experiments that CHOTA is sensitive to all relevant errors and provides a continuous, interpretable scale. The work includes a thorough metric survey, an efficient quasi-linear computation, and publicly available Python code, offering a robust tool for fair, holistic evaluation and potentially guiding the design of globally coherent cell-tracking methods with meaningful biological relevance.

Abstract

The evaluation of cell tracking results steers the development of tracking methods, significantly impacting biomedical research. This is quantitatively achieved by means of evaluation metrics. Unfortunately, current metrics favor local correctness and weakly reward global coherence, impeding high-level biological analysis. To also foster global coherence, we propose the CHOTA metric (Cell-specific Higher Order Tracking Accuracy) which unifies the evaluation of all relevant aspects of cell tracking: cell detections and local associations, global coherence, and lineage tracking. We achieve this by introducing a new definition of the term 'trajectory' that includes the entire cell lineage and by including this into the well-established HOTA metric from general multiple object tracking. Furthermore, we provide a detailed survey of contemporary cell tracking metrics to compare our novel CHOTA metric and to show its advantages. All metrics are extensively evaluated on state-of-the-art real-data cell tracking results and synthetic results that simulate specific tracking errors. We show that CHOTA is sensitive to all tracking errors and gives a good indication of the biologically relevant capability of a method to reconstruct the full lineage of cells. It introduces a robust and comprehensive alternative to the currently used metrics in cell tracking. Python code is available at https://github.com/CellTrackingChallenge/py-ctcmetrics .

Figures (22)

• Figure 1: Sensitivity of selected cell tracking metrics to specific tracking errors, measured by the correlation in our experiments. Error types are identity switches (IDSw), false (FP) or missing (FN) detections, unmatched detection-annotation pairs (Match), and missed mitosis detections (Mitosis). From left to right, we present the CTC celltrackingchallenge, CTMC ctmc with HOTA luiten2020IJCV, and biological metrics (BIO) together with CHOTA. Some metrics are less sensitive to a particular type of error (correlation is $\ll 1.0$), or even not sensitive at all ($= 0.0$; e.g. CTMC metrics to mitosis detection errors). CHOTA and TRA are sensitive enough to all tracking issues, but CHOTA is the only metric that also rates individual errors by their global impact. Thus, the same amount of IDSw (or Mitosis) errors may influence variously (depending on how much the randomly inserted errors cancel out themselves) the global coherence, which acts as noise to the correlation and is actually visible in the correlation $< 1.0$.
• Figure 2: The common ID-oriented and our lineage-oriented trajectory. A node represents a cell in a specific frame with its respective ID and an edge emphasizes the evolution over time. Each line shows the same lineage tree but highlights a different trajectory (red) for a specific cell detection (black). Our definition accounts for ID-oriented information and also biologically relevant relations to ancestors and descendants.
• Figure 3: An example prediction (green) to annotation (yellow) mapping presenting the relations to calculate $\mathcal{A}^\sigma(c)$ from Equation \ref{['eq:assoc']} for two matched pairs $c$ (black). Using our lineage-oriented trajectory, TPA, FPA, and FNA reflect inter- and intra-ID relations. On the left, the ground truth annotation of $c$ relates to two daughter cells of which only one is represented in the prediction, leading to a low $\mathcal{A}^\sigma(c)=\frac{7}{15}$. On the right, the lineage of $c$ is almost correct, leading to a higher $\mathcal{A}^\sigma(c)=\frac{6}{8}$.
• Figure 4: The behavior of metrics if specific tracking errors are induced into a perfect tracking result (BF-C2DL-HSC, Seq. 01). The biological metrics (CT, TF, BC, CCA) are not equally treating similar errors, visible in the large variances. TRA is not adequately reflecting the main tracking issues of ID switches and missed mitosis detections, apparent in consistent values close to 1. CHOTA is the only metric that is both, equal and continuously exploiting a large value range when tracking errors are induced.
• Figure 5: Synthetic error induction for Dataset BF-C2DL-HSC (Sequence 01)