TreeFormer: Single-view Plant Skeleton Estimation via Tree-constrained Graph Generation

TL;DR

This work presents TreeFormer, a plant skeleton estimator via tree-constrained graph generation, which combines learning-based graph generation with traditional graph algorithms to impose the constraints during the training loop.

Abstract

Accurate estimation of plant skeletal structure (e.g., branching structure) from images is essential for smart agriculture and plant science. Unlike human skeletons with fixed topology, plant skeleton estimation presents a unique challenge, i.e., estimating arbitrary tree graphs from images. While recent graph generation methods successfully infer thin structures from images, it is challenging to constrain the output graph strictly to a tree structure. To this problem, we present TreeFormer, a plant skeleton estimator via tree-constrained graph generation. Our approach combines learning-based graph generation with traditional graph algorithms to impose the constraints during the training loop. Specifically, our method projects an unconstrained graph onto a minimum spanning tree (MST) during the training loop and incorporates this prior knowledge into the gradient descent optimization by suppressing unwanted feature values. Experiments show that our method accurately estimates target plant skeletal structures for multiple domains: Synthetic tree patterns, real botanical roots, and grapevine branches. Our implementations are available at https://github.com/huntorochi/TreeFormer/.

Figures (15)

• Figure 1: We propose a method for single-image plant skeleton estimation combining learning-based graph generators with traditional graph algorithm (i.e., MST). The red lines show the predicted graph edges. Compared to (a) an unconstrained graph generator and (b) a naive tree-graph constraint implementation, (c) our method naturally imposes the constraint during the graph generation models' training. Our method can be directly applied to plant science and agricultural applications, such as (d) time-series reconstruction of botanical roots.
• Figure 2: Overview of reparameterization layer that can be easily plugged into off-the-shelf graph generators. Given unconstrained edge predictions by graph generators, our method projects it to the closest constrained graph (i.e., tree) using a non-differentiable MST algorithm. Comparing constrained and unconstrained edges, unwanted edge features are selectively suppressed so that the graph becomes the tree.
• Figure 3: Example images from the dataset we used for our experiments. Annotated graphs are superimposed. Yellow dots and red lines indicate nodes and edges.
• Figure 4: Visual results for the synthetic tree pattern dataset. From left to right: Input images, results of the two-stage method (similar to Guyot), the unconstrained method (identical to RelationFormer Relationformer), a naive implementation with test-time constraint, and ours are shown. We translucently overlay the estimated and ground truth edges with red and blue lines, respectively. While all methods accurately detect nodes, only our method accurately predicts the availability of edges from given images compared to the baseline methods.
• Figure 5: Visual results for the real image datasets. Red lines, yellow dots, and cyan dots indicate the predicted graph edges, nodes, and keypoints (i.e., joints and leaf nodes). Our method accurately estimates the target plant structures compared with baseline methods, demonstrating the applicability of our method for practical uses in plant science and agriculture.