Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visual-Language-Guided Task Planning for Horticultural Robots

Jose Cuaran, Kendall Koe, Aditya Potnis, Naveen Kumar Uppalapati, Girish Chowdhary

TL;DR

This work tackles high-level task planning for horticultural robots by integrating a vision-language model with perception, planning, and control modules. It introduces a modular VLM-guided planning framework and an open-vocabulary semantic occupancy mapping pipeline, evaluated on a Gazebo-based benchmark spanning monoculture and polyculture scenarios. Results show strong performance of zero-shot VLM planning on simple, short-horizon tasks but substantial degradation for long-horizon, multi-plant tasks, with map noise and limited spatial grounding as key bottlenecks. The study highlights the potential of visual-language grounding in agricultural robotics while identifying critical areas—spatial reasoning, memory management, and map robustness—that require further development for field deployment.

Abstract

Crop monitoring is essential for precision agriculture, but current systems lack high-level reasoning. We introduce a novel, modular framework that uses a Visual Language Model (VLM) to guide robotic task planning, interleaving input queries with action primitives. We contribute a comprehensive benchmark for short- and long-horizon crop monitoring tasks in monoculture and polyculture environments. Our main results show that VLMs perform robustly for short-horizon tasks (comparable to human success), but exhibit significant performance degradation in challenging long-horizon tasks. Critically, the system fails when relying on noisy semantic maps, demonstrating a key limitation in current VLM context grounding for sustained robotic operations. This work offers a deployable framework and critical insights into VLM capabilities and shortcomings for complex agricultural robotics.

Visual-Language-Guided Task Planning for Horticultural Robots

TL;DR

This work tackles high-level task planning for horticultural robots by integrating a vision-language model with perception, planning, and control modules. It introduces a modular VLM-guided planning framework and an open-vocabulary semantic occupancy mapping pipeline, evaluated on a Gazebo-based benchmark spanning monoculture and polyculture scenarios. Results show strong performance of zero-shot VLM planning on simple, short-horizon tasks but substantial degradation for long-horizon, multi-plant tasks, with map noise and limited spatial grounding as key bottlenecks. The study highlights the potential of visual-language grounding in agricultural robotics while identifying critical areas—spatial reasoning, memory management, and map robustness—that require further development for field deployment.

Abstract

Crop monitoring is essential for precision agriculture, but current systems lack high-level reasoning. We introduce a novel, modular framework that uses a Visual Language Model (VLM) to guide robotic task planning, interleaving input queries with action primitives. We contribute a comprehensive benchmark for short- and long-horizon crop monitoring tasks in monoculture and polyculture environments. Our main results show that VLMs perform robustly for short-horizon tasks (comparable to human success), but exhibit significant performance degradation in challenging long-horizon tasks. Critically, the system fails when relying on noisy semantic maps, demonstrating a key limitation in current VLM context grounding for sustained robotic operations. This work offers a deployable framework and critical insights into VLM capabilities and shortcomings for complex agricultural robotics.
Paper Structure (17 sections, 3 equations, 4 figures, 6 tables)

This paper contains 17 sections, 3 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Methods
  4. Perception
  5. Planning
  6. Control
  7. Experiments
  8. Simulation Environment
  9. Metrics
  10. Benchmark Description
  11. Ablation Studies
  12. Results and Discussion
  13. Main Results
  14. Ablation Results
  15. Failure Cases
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Sample episode of a single-plant, single-target monitoring task executed by the VLM agent. (a) The VLM takes as input the task prompt, the context and tools description. (b) The navigation tool call is used to navigate to the goal object and (c) The manipulation tools are invoked at different time steps until completing the inspection task.
  • Figure 2: Overall System Pipeline Given context and the assigned task, the agricultural agent selects from a library of action primitives to achieve the task. The agent can switch between manipulation and navigation modes. A library of primitive tasks are available for the agent to choose from to accomplish the task. Once the task is complete or the agent requires help, a human can be prompted to intervene.
  • Figure 3: Semantic mapping pipeline. The RGBD data is used to generate an occupancy map of the environment. Detic is used for open vocabulary object detection and segmentation. The output is backprojected and filtered with kalman filtering. Both branches are projected to create a semantic occupancy map.
  • Figure 4: Simulation Environments Three gazebo simulation environments are used to evaluate the system. (a) A monoculture environment filled with various ripe and unripe tomatoes. (b) A polyculture environment filled with tomatoes, orange peppers, red peppers, and eggplants. (c) A polyculture environments filled with tomatoes, green peppers, red peppers, and eggplants as well as lettuce and strawberries on raised tables.