Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SUM-AgriVLN: Spatial Understanding Memory for Agricultural Vision-and-Language Navigation

Xiaobei Zhao, Xingqi Lyu, Xiang Li

TL;DR

SUM-AgriVLN addresses the limitation of independent-episode VLN in agriculture by adding Spatial Understanding Memory (SUM) that reconstructs 3D scene geometry and stores frontal and oblique memory representations for recall. The SUM module is integrated into the AgriVLN base model to provide global scene context, guiding low-level action decisions using both language and visual inputs. On the A2A benchmark, SUM-AgriVLN improves the overall Success Rate from 0.47 to 0.54 with a modest increase in Navigation Error to 2.93 m, establishing state-of-the-art performance in agricultural VLN and enabling more reliable autonomous farming assistance. Ablation and qualitative analyses demonstrate the effectiveness and tradeoffs of different memory perspectives and task complexities, while also highlighting areas for future work in dynamic scenes and richer memory representations.

Abstract

Agricultural robots are emerging as powerful assistants across a wide range of agricultural tasks, nevertheless, still heavily rely on manual operation or fixed rail systems for movement. The AgriVLN method and the A2A benchmark pioneeringly extend Vision-and-Language Navigation (VLN) to the agricultural domain, enabling robots to navigate to the target positions following the natural language instructions. In practical agricultural scenarios, navigation instructions often repeatedly occur, yet AgriVLN treat each instruction as an independent episode, overlooking the potential of past experiences to provide spatial context for subsequent ones. To bridge this gap, we propose the method of Spatial Understanding Memory for Agricultural Vision-and-Language Navigation (SUM-AgriVLN), in which the SUM module employs spatial understanding and save spatial memory through 3D reconstruction and representation. When evaluated on the A2A benchmark, our SUM-AgriVLN effectively improves Success Rate from 0.47 to 0.54 with slight sacrifice on Navigation Error from 2.91m to 2.93m, demonstrating the state-of-the-art performance in the agricultural domain. Code: https://github.com/AlexTraveling/SUM-AgriVLN.

SUM-AgriVLN: Spatial Understanding Memory for Agricultural Vision-and-Language Navigation

TL;DR

SUM-AgriVLN addresses the limitation of independent-episode VLN in agriculture by adding Spatial Understanding Memory (SUM) that reconstructs 3D scene geometry and stores frontal and oblique memory representations for recall. The SUM module is integrated into the AgriVLN base model to provide global scene context, guiding low-level action decisions using both language and visual inputs. On the A2A benchmark, SUM-AgriVLN improves the overall Success Rate from 0.47 to 0.54 with a modest increase in Navigation Error to 2.93 m, establishing state-of-the-art performance in agricultural VLN and enabling more reliable autonomous farming assistance. Ablation and qualitative analyses demonstrate the effectiveness and tradeoffs of different memory perspectives and task complexities, while also highlighting areas for future work in dynamic scenes and richer memory representations.

Abstract

Agricultural robots are emerging as powerful assistants across a wide range of agricultural tasks, nevertheless, still heavily rely on manual operation or fixed rail systems for movement. The AgriVLN method and the A2A benchmark pioneeringly extend Vision-and-Language Navigation (VLN) to the agricultural domain, enabling robots to navigate to the target positions following the natural language instructions. In practical agricultural scenarios, navigation instructions often repeatedly occur, yet AgriVLN treat each instruction as an independent episode, overlooking the potential of past experiences to provide spatial context for subsequent ones. To bridge this gap, we propose the method of Spatial Understanding Memory for Agricultural Vision-and-Language Navigation (SUM-AgriVLN), in which the SUM module employs spatial understanding and save spatial memory through 3D reconstruction and representation. When evaluated on the A2A benchmark, our SUM-AgriVLN effectively improves Success Rate from 0.47 to 0.54 with slight sacrifice on Navigation Error from 2.91m to 2.93m, demonstrating the state-of-the-art performance in the agricultural domain. Code: https://github.com/AlexTraveling/SUM-AgriVLN.
Paper Structure (21 sections, 4 equations, 5 figures, 3 tables)

This paper contains 21 sections, 4 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Agricultural Vision-and-Language Navigation
  4. Spatial Memory in Vision-and-Language Navigation
  5. 3D Reconstruction
  6. Methodology
  7. Task Definition
  8. Spatial Understanding Memory
  9. Spatial Understanding
  10. Spatial Memory
  11. Base Model
  12. Experiments
  13. Experimental Setting
  14. Evaluation Metrics
  15. Qualitative Experiment
  16. ...and 6 more sections

Figures (5)

  • Figure 1: SUM-AgriVLN v.s. standard agricultural VLN methods on a simple example.
  • Figure 2: SUM-AgriVLN methodology illustration: The purple and blue parts represent the Spatial Understanding and Spatial Memory processes, respectively. The base model leverages the spatial memory to recall the scene, then understands the linguistic and visual inputs to reason the low-level action sequence.
  • Figure 3: Qualitative experiment illustration: The grey and purple message boxes represent the reasoning thoughts of baseline and SUM-AgriVLN, respectively.
  • Figure 4: Ablation experiment illustration: (a) and (b) are SUM-AgriVLN with frontal v.s. oblique Spatial Memory on different scene classifications. (c) and (d) are SUM-AgriVLN v.s. SUM-AgriVLN $_{w/o}$ SUM on different task complexity.
  • Figure 5: SUM-AgriVLN prompt template: The system prompt and the user prompt are presented in the purple and blue boxes, respectively. The prompt template for the module of Subtask List is available in AgriVLN arXiv:AgriVLN. (Zoom in for better observation)