Fine-Tuned Vision Transformers Capture Complex Wheat Spike Morphology for Volume Estimation from RGB Images

TL;DR

The paper tackles non-destructive estimation of wheat spike volume from RGB images, a challenging task due to depth loss, projection effects, and irregular spike geometry. It develops a large, diverse dataset with ground-truth volumes from structured-light scans and rigorously compares area-based and geometric baselines against neural-network models based on Vision Transformers (DINOv2/DINOv3) with MLP/LSTM/Transformer heads, exploring both frozen and fine-tuned backbones and multi-view fusion. Key findings show that fine-tuned DINOv3 with an MLP downstream yields the best indoor six-view accuracy (Corr ≈ 0.97, MAPE ≈ 4.67%), and field-adapted single-view models still achieve strong performance (Corr ≈ 0.90, MAPE ≈ 8.39%), while object shape significantly influences baseline methods. The study demonstrates a fast, non-destructive pipeline for wheat spike volume phenotyping, introduces the concept of fruiting capacity as summed volume per area, and provides practical models that generalize from controlled indoor setups to field conditions, enabling scalable phenotyping for breeding and crop-yield analyses.

Abstract

Estimating three-dimensional morphological traits such as volume from two-dimensional RGB images presents inherent challenges due to the loss of depth information, projection distortions, and occlusions under field conditions. In this work, we explore multiple approaches for non-destructive volume estimation of wheat spikes using RGB images and structured-light 3D scans as ground truth references. Wheat spike volume is promising for phenotyping as it shows high correlation with spike dry weight, a key component of fruiting efficiency. Accounting for the complex geometry of the spikes, we compare different neural network approaches for volume estimation from 2D images and benchmark them against two conventional baselines: a 2D area-based projection and a geometric reconstruction using axis-aligned cross-sections. Fine-tuned Vision Transformers (DINOv2 and DINOv3) with MLPs achieve the lowest MAPE of 5.08\% and 4.67\% and the highest correlation of 0.96 and 0.97 on six-view indoor images, outperforming fine-tuned CNNs (ResNet18 and ResNet50), wheat-specific backbones, and both baselines. When using frozen DINO backbones, deep-supervised LSTMs outperform MLPs, whereas after fine-tuning, improved high-level representations allow simple MLPs to outperform LSTMs. We demonstrate that object shape significantly impacts volume estimation accuracy, with irregular geometries such as wheat spikes posing greater challenges for geometric methods than for deep learning approaches. Fine-tuning DINOv3 on field-based single side-view images yields a MAPE of 8.39\% and a correlation of 0.90, providing a novel pipeline and a fast, accurate, and non-destructive approach for wheat spike volume phenotyping.

Figures (19)

• Figure 1: Overview of the study design separated in three sections data acquisition, pre-processing, and volume estimation. In the data acquisition step, wheat spikes were sampled, imaged, and scanned. Images were pre-processed to get the binary segmentations masks. The scans were processed and the ground truth volume was extracted. Two baselines and neural networks were used to estimate wheat spike volume based on images.
• Figure 1: Linear, quadratic, and exponential curve fitted to the number of spike pixels of the area baseline and the measured volume of the training dataset (grey), and their respective $R^2$ score. Test dataset (blue) was added for visualization purpose. Separate curves were fitted for estimations based on one image (a), two images (b), four images(c), and six images (d).
• Figure 2: Overview of the camera setup and image acquisition process. The imaging setup with a 20 cm distance between the camera lens and a blue background panel (a). Representative images of a side, frontal, and oblique view of the wheat spike captured indoors (b). Representative image of a side view captured in the field (c).
• Figure 2: Linear, quadratic, and exponential curve fitted to the estimated spikes based on the geometric baseline and the measured volume of the training dataset (grey), and their respective $R^2$ score. Test dataset (blue) was added for visualization purpose. Separate curves were fitted for estimations based on one image (a), two images (b), four images(c), and six images (d).
• Figure 3: Overview of preprocessing steps, including an original image (a), after bar removal and inpainting (b), and after segmentation (c).