Push Past Green: Learning to Look Behind Plant Foliage by Moving It

TL;DR

Self-supervision is used to train SRPNet, a neural network that predicts what space is revealed on execution of a candidate action on a given plant, and this method is used with the cross-entropy method to predict actions that are effective at revealing space beneath plant foliage.

Abstract

Autonomous agriculture applications (e.g., inspection, phenotyping, plucking fruits) require manipulating the plant foliage to look behind the leaves and the branches. Partial visibility, extreme clutter, thin structures, and unknown geometry and dynamics for plants make such manipulation challenging. We tackle these challenges through data-driven methods. We use self-supervision to train SRPNet, a neural network that predicts what space is revealed on execution of a candidate action on a given plant. We use SRPNet with the cross-entropy method to predict actions that are effective at revealing space beneath plant foliage. Furthermore, as SRPNet does not just predict how much space is revealed but also where it is revealed, we can execute a sequence of actions that incrementally reveal more and more space beneath the plant foliage. We experiment with a synthetic (vines) and a real plant (Dracaena) on a physical test-bed across 5 settings including 2 settings that test generalization to novel plant configurations. Our experiments reveal the effectiveness of our overall method, PPG, over a competitive hand-crafted exploration method, and the effectiveness of SRPNet over a hand-crafted dynamics model and relevant ablations.

Figures (17)

• Figure 1: (left) Plants self-occlude themselves. Two examples of leaves and branches being pushed aside for inspection and picking fruits. This paper develops learning algorithms that enable robots to tackle this plant self-occlusion problem. We show actions executed by the robot to expose the space behind vines (middle) and Dracaena plant (right).
• Figure 1: Average precision for different models at predicting space revealed. Higher is better. Our proposed input representation outperforms simpler alternatives and data augmentation boosts performance.
• Figure 2: Overview of PushPastGreen. PushPastGreen learns to manipulate plants to reveal the space behind them thus tackling the plant self-occlusion problem. PushPastGreen includes Space-Revealed Prediction Network (SRPNet) that predicts where space is revealed upon execution of a pushing action, as shown in (b) and described in Sec. \ref{['sec:dummy-forward-model']}. SRPNet can not only rank actions based on how much space they will reveal, but because it can also predict where space gets revealed, it can also be used for executing multi-step trajectories that explore all the space behind the vines as shown in (a) and described in Sec. \ref{['sec:dummy-control']}. SRPNet is trained using self-supervision as described in Sec. \ref{['sec:dummy-data-collection']}.
• Figure 2: PPG selected actions are more effective at revealing space.
• Figure 3: Hardware setup for vines (left) and real Dracaena plant (right). We use a grabber as the end-effector song2020graspingyoung2020visual. View from the RGB-D camera is in the inset. The task is to move the vines and the Dracaena leaves aside to reveal the space occluded by them.