ZS-TreeSeg: A Zero-Shot Framework for Tree Crown Instance Segmentation

TL;DR

ZS-TreeSeg addresses tree crown instance segmentation in dense canopies under zero-shot constraints by combining mature canopy semantic segmentation with a flow-guided, gradient-based instance separation that models crowns as star-convex objects. The method constrains the search space with a semantic prior $M_{\text{sem}}$ and uses a gradient flow $\mathbf{V}$ to separate touching crowns without per-instance training, via an inference procedure that updates pixel positions with $p_{\tau+1}=p_{\tau}+\mathbf{V}(p_{\tau})$ until convergence to sink points. Evaluations on NEON and BAMFORESTS show competitive quantitative performance (e.g., $mAP@50$ of 42.30% and 67.31%, respectively) against some supervised baselines, and qualitative analyses demonstrate robust zero-shot generalization across scales and canopy densities. The work discusses practical considerations like data standardization and introduces a physical prior—the average crown diameter $D$—to control segmentation granularity, offering a scalable, training-free tool for forest monitoring and label generation.

Abstract

Individual tree crown segmentation is an important task in remote sensing for forest biomass estimation and ecological monitoring. However, accurate delineation in dense, overlapping canopies remains a bottleneck. While supervised deep learning methods suffer from high annotation costs and limited generalization, emerging foundation models (e.g., Segment Anything Model) often lack domain knowledge, leading to under-segmentation in dense clusters. To bridge this gap, we propose ZS-TreeSeg, a Zero-Shot framework that adapts from two mature tasks: 1) Canopy Semantic segmentation; and 2) Cells instance segmentation. By modeling tree crowns as star-convex objects within a topological flow field using Cellpose-SAM, the ZS-TreeSeg framework forces the mathematical separation of touching tree crown instances based on vector convergence. Experiments on the NEON and BAMFOREST datasets and visual inspection demonstrate that our framework generalizes robustly across diverse sensor types and canopy densities, which can offer a training-free solution for tree crown instance segmentation and labels generation.

Figures (8)

• Figure 1: Overview of the Zero-Shot Tree Crown Segmentation Framework. The pipeline integrates a semantic prior to extract the canopy mask, followed by a cell segmentation model (Cellpose-SAM) that predicts gradient vector fields to isolate individual tree instances.
• Figure 2: Effect of semantic masking on zero-shot instance segmentation. (Top) Results without semantic filtering, showing erroneous detections over background textures. (Bottom) Results with semantic prior ($M_{\text{sem}}$), where instance predictions are constrained to canopy regions.
• Figure 3: Conceptual framework of the proposed flow-based segmentation. (1) Feature Encoding: A ViT backbone extracts a latent representation $\mathbf{Z}$ capturing global shape context. (2) Flow Prediction: The network predicts a unit vector field $\mathbf{V}$ (gradient of a diffusion potential) that directs pixels toward object interiors. (3) Grouping: Segmentation is achieved by iteratively tracking pixel trajectories until they converge to stable sinks.
• Figure 4: Results of the zero-shot instance segmentation. The model demonstrates robust generalization across different scales and species, accurately delineating both large isolated crowns and smaller, clustered trees.
• Figure 5: Performance comparison across varying canopy densities. While the model achieves near-perfect segmentation in sparse urban contexts (top), it also maintains high separability in dense, overlapping canopy environments (bottom) by leveraging flow field dynamics.