ST-DETrack: Identity-Preserving Branch Tracking in Entangled Plant Canopies via Dual Spatiotemporal Evidence

TL;DR

ST-DETrack tackles the challenge of maintaining branch identity in entangled plant canopies over time. It introduces a spatiotemporal fusion framework with a spatial decoder and a temporal decoder, coupled by adaptive gating and a negative gravitropism prior. On Brassica napus, it achieves 93.6% Branch Matching Accuracy, outperforming static segmentation and generic MOT trackers. The method enables robust extraction of dynamic phenotypic traits and supports high-throughput plant science applications.

Abstract

Automated extraction of individual plant branches from time-series imagery is essential for high-throughput phenotyping, yet it remains computationally challenging due to non-rigid growth dynamics and severe identity fragmentation within entangled canopies. To overcome these stage-dependent ambiguities, we propose ST-DETrack, a spatiotemporal-fusion dual-decoder network designed to preserve branch identity from budding to flowering. Our architecture integrates a spatial decoder, which leverages geometric priors such as position and angle for early-stage tracking, with a temporal decoder that exploits motion consistency to resolve late-stage occlusions. Crucially, an adaptive gating mechanism dynamically shifts reliance between these spatial and temporal cues, while a biological constraint based on negative gravitropism mitigates vertical growth ambiguities. Validated on a Brassica napus dataset, ST-DETrack achieves a Branch Matching Accuracy (BMA) of 93.6%, significantly outperforming spatial and temporal baselines by 28.9 and 3.3 percentage points, respectively. These results demonstrate the method's robustness in maintaining long-term identity consistency amidst complex, dynamic plant architectures.

Figures (5)

• Figure 1: Morphological evolution of rapeseed branching architecture across growth stages. Spatial ambiguity increases sharply as branches elongate and intertwine, necessitating temporal evidence for reliable identity assignment.
• Figure 2: Overall dual-decoder fusion architecture comprising a shared encoder, spatial decoder, temporal decoder, and adaptive fusion mechanism.
• Figure 3: The annotation system interface demonstrating order-based semantic matching and SIFT-based adaptive calibration. The system enables efficient tracking of branch junctions across time-series frames through intelligent workflow design and computer-vision-assisted algorithms.
• Figure 4: Qualitative comparison of instance segmentation models on branch-level predictions. Each row shows predictions from a different model (Ground Truth, SOLOv2, Mask2Former, MaskDINO, YOLOv12-seg, YOLACT, and ST-DETrack) for the same plant sample. Columns represent individual branches (Branch 1--5) with their corresponding IoU scores displayed below each prediction. Color-coded masks indicate different branch instances. Ground truth annotations (top row) serve as the reference for evaluation. Note that while SOLOv2, Mask2Former, and YOLOv12-seg achieve relatively high IoU scores on well-separated branches, they struggle with fine-grained boundary delineation. MaskDINO and YOLACT show inconsistent performance with lower IoU values, particularly on occluded or thin branches. ST-DETrack (bottom row) demonstrates superior segmentation quality across all branches by leveraging spatiotemporal evidence, achieving more accurate boundary localization and better handling of challenging cases such as inter-branch occlusions and morphological variations. The "???" markers indicate failed detections where the model completely missed the branch instance.
• Figure 5: Longitudinal branch identity tracking visualization across five representative test plants. Each row shows a temporal sequence (left to right: progressive time steps) for a single plant with color-coded branch IDs maintained consistently across frames. From top to bottom: (a) BR017-028111 and (b) BR017-028112 demonstrate early-stage sparse configurations with well-separated branches; (c) BR017-028121 and (d) BR017-028122 show mid-stage transitions with increasing spatial overlap; (e) BR017-028213 exhibits late-stage dense canopy with severe inter-branch occlusions. Individual primary branches retain unique colors throughout their development, validating the spatiotemporal fusion mechanism's ability to handle diverse structural complexity and growth stages. The horizontal arrangement within each row demonstrates temporal consistency, where identical colors across time steps confirm successful identity preservation.