Botany-Bot: Digital Twin Monitoring of Occluded and Underleaf Plant Structures with Gaussian Splats

TL;DR

Botany-Bot is a system for building detailed “annotated digital twins” of living plants using two stereo cameras, a digital turntable inside a lightbox, an industrial robot arm, and 3D segmentated Gaussian Splat models to take high-resolution indexable images of occluded details.

Abstract

Commercial plant phenotyping systems using fixed cameras cannot perceive many plant details due to leaf occlusion. In this paper, we present Botany-Bot, a system for building detailed "annotated digital twins" of living plants using two stereo cameras, a digital turntable inside a lightbox, an industrial robot arm, and 3D segmentated Gaussian Splat models. We also present robot algorithms for manipulating leaves to take high-resolution indexable images of occluded details such as stem buds and the underside/topside of leaves. Results from experiments suggest that Botany-Bot can segment leaves with 90.8% accuracy, detect leaves with 86.2% accuracy, lift/push leaves with 77.9% accuracy, and take detailed overside/underside images with 77.3% accuracy. Code, videos, and datasets are available at https://berkeleyautomation.github.io/Botany-Bot/.

Figures (5)

• Figure 1: Botany-Bot creates detailed 3D digital twins of plants with a turntable and fixed cameras, segmenting them into individual components. Botany-Bot then augments them by lifting or pushing down individual leaves to capture high-resolution images of the occluded sides of leaves. Botany-Bot can also compute plant metrics; for example, the highlighted leaf has a leaf area of 25.6$\text{cm}^2$, leaf length of 7.2cm, and ground height of 27.2cm.
• Figure 2: Annotated 3D Digital Twins: Botany-Bot produces high-fidelity 3D plant models along with segmented leaves (middle). By inspecting individual leaves, these models can be used to obtain physical properties like leaf height and area. In addition, the model is enhanced with robot interaction data where a robot arm lifts each leaf and examines its underside in high resolution. Pictured are examples of two leaves.
• Figure 3: Botany-Bot Plant Scan, Inspection & Interaction System. Multi-view data collection is contained within a lightbox (left) which ensures uniform directional illumination for NeRF reconstruction. (Middle) a view inside the lightbox with turntable, plant, and 2 cameras. (right) the robot setup used for rotating the plant, lifting/pushing leaves, and imaging the undersides/tops of plant. The robot uses a digital turntable to minimize occlusion while manipulating the plant.
• Figure 4: Software Diagram for Botany-Bot. The modeling component takes plant images from multiple views, and models the plants by constructing in 3D, segmenting individual leaves and measuring the characteristics. Then the inspection component plans the leaf lifting, evaluates view coverage and we create annotated digital twins.
• Figure 5: Botany-Bot Segmented Digital Twins: Presented here are a side view and top view for six different plant species. Each cell shows two different viewpoints of the 3D model, a side and top view. The RGB rendering is provided next to the colorized segmentation of plant components, each color corresponding to a different segment. Note both the visual fidelity of the models as well as the fine-grained leaf 3D segmentations.