Robotic Monitoring of Colorimetric Leaf Sensors for Precision Agriculture

TL;DR

The paper addresses the need for direct, real-time crop-stress indicators by introducing a co-designed system that couples passive colorimetric metasurface leaf sensors with a lightweight autonomous rover equipped for RGB detection and hyperspectral imaging. A YOLOv7-based detector localizes leaf sensors, and a motorized-mirror hyperspectral system performs targeted resonance measurements, enabling high-resolution spectral data collection at field scale. Key contributions include outdoor resonance measurements from passive leaf sensors, a low-SWaP robot platform (beast) integrated with hyperspectral imaging, and an end-to-end vision-sensing pipeline for autonomous data collection. The study demonstrates resonances near 654–655 nm under varied indoor/outdoor conditions, with up to ~80% success in resonance acquisition, indicating practical potential for precision agriculture. This work paves the way for scalable, direct crop-health monitoring that can inform targeted farming interventions.

Abstract

Common remote sensing modalities (RGB, multispectral, hyperspectral imaging or LiDAR) are often used to indirectly measure crop health and do not directly capture plant stress indicators. Commercially available direct leaf sensors are bulky, powered electronics that are expensive and interfere with crop growth. In contrast, low-cost, passive and bio-degradable leaf sensors offer an opportunity to advance real-time monitoring as they directly interface with the crop surface while not interfering with crop growth. To this end, we co-design a sensor-detector system, where the sensor is a passive colorimetric leaf sensor that directly measures crop health in a precision agriculture setting, and the detector autonomously obtains optical signals from these leaf sensors. The detector comprises a low size weight and power (SWaP) mobile ground robot with an onboard monocular RGB camera and object detector to localize each leaf sensor, as well as a hyperspectral camera with a motorized mirror and halogen light to acquire hyperspectral images. The sensor's crop health-dependent optical signals can be extracted from the hyperspectral images. The proof-of-concept system is demonstrated in row-crop environments both indoors and outdoors where it is able to autonomously navigate, locate and obtain a hyperspectral image of all leaf sensors present, and acquire interpretable spectral resonance with 80 $\%$ accuracy within a required retrieval distance from the sensor.

Figures (9)

• Figure 1: Our ground robot platform, beast, is equipped with visual-inertial odometry-based localization and an RGB monocular camera to localize colorimetric leaf sensors mounted on plants. Hyperspectral images and spectra of the leaf sensors are acquired via an on-board hyperspectral camera and a motorized mirror control system.
• Figure 2: Our system consists of a metasurface colorimetric leaf sensor; ground robot; and hyperspectral imaging submodule.
• Figure 3: top left A schematic of the metasurface sensor architecture with $TiO_2$ nanopillars and waveguide layer. top right A scanning electron micrograph of the metasurface which shows the circular nanopillars are 200nm in diameter on a 400nm pitch. bottom A schematic of the co-designed sensor-detector system in outdoor row crop environments.
• Figure 4: An overview of the beast platform (left) and a close up of hyperspectral sub-module (right). (1) FLIR Chameleon3 RGB camera with pan-tilt mechanism, (2) NVIDIA Jetson Xavier NX, (3) hyperspectral sub-module, (4) Thorlabs SLS201L(/M) tungsten-halogen light source, (5) Stereolabs Zed 2i VIO, (6) Optotune Fast Steering Mirror MR-15-30-PS $15$mm, (7) $70$mm focal length lens, (8) Headwall Nano-Hyperspec VNIR Imaging Sensor, with spatial resolution of $640$p, $400-1000$nm wavelength, and spectral resolution with $270$ bands.
• Figure 5: Experiments were conducted in three different settings: (top) indoors and controlled, (middle) outdoors and unstructured, and (bottom) outdoors and structured. These experiments included trials across different terrains (gravel, soil), and across five different days with varying lighting conditions (overcast, partial cloud coverage, and sunny).