Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HarvestFlex: Strawberry Harvesting via Vision-Language-Action Policy Adaptation in the Wild

Ziyang Zhao, Shuheng Wang, Zhonghua Miao, Ya Xiong

TL;DR

This work built an end-to-end closed-loop system on the HarvestFlex platform using three-view RGB sensing and intentionally avoided depth clouds and explicit geometric calibration and showed non-trivial closed-loop picking with fewer than four hours of real data, while remaining limited by close-range observability loss and contact-dynamics mismatch.

Abstract

This work presents the first study on transferring vision-language-action (VLA) policies to real greenhouse tabletop strawberry harvesting, a long-horizon, unstructured task challenged by occlusion and specular reflections. We built an end-to-end closed-loop system on the HarvestFlex platform using three-view RGB sensing (two fixed scene views plus a wrist-mounted view) and intentionally avoided depth clouds and explicit geometric calibration. We collected 3.71 h of VR teleoperated demonstrations (227 episodes) and fine-tuned pi_0, pi_0.5, and WALL-OSS with full fine-tuning and LoRA. Under a unified 50 trials real-greenhouse protocol and metrics spanning completion, pi_0.5 with full fine-tuning achieved success rate of 74.0% with 32.6 s/pick and damage rate of 4.1%. Asynchronous inference-control decoupling further improved performance over synchronous deployment. Results showed non-trivial closed-loop picking with fewer than four hours of real data, while remaining limited by close-range observability loss and contact-dynamics mismatch. A demonstration video is available at: https://youtu.be/bN8ZowZKPMI.

HarvestFlex: Strawberry Harvesting via Vision-Language-Action Policy Adaptation in the Wild

TL;DR

This work built an end-to-end closed-loop system on the HarvestFlex platform using three-view RGB sensing and intentionally avoided depth clouds and explicit geometric calibration and showed non-trivial closed-loop picking with fewer than four hours of real data, while remaining limited by close-range observability loss and contact-dynamics mismatch.

Abstract

This work presents the first study on transferring vision-language-action (VLA) policies to real greenhouse tabletop strawberry harvesting, a long-horizon, unstructured task challenged by occlusion and specular reflections. We built an end-to-end closed-loop system on the HarvestFlex platform using three-view RGB sensing (two fixed scene views plus a wrist-mounted view) and intentionally avoided depth clouds and explicit geometric calibration. We collected 3.71 h of VR teleoperated demonstrations (227 episodes) and fine-tuned pi_0, pi_0.5, and WALL-OSS with full fine-tuning and LoRA. Under a unified 50 trials real-greenhouse protocol and metrics spanning completion, pi_0.5 with full fine-tuning achieved success rate of 74.0% with 32.6 s/pick and damage rate of 4.1%. Asynchronous inference-control decoupling further improved performance over synchronous deployment. Results showed non-trivial closed-loop picking with fewer than four hours of real data, while remaining limited by close-range observability loss and contact-dynamics mismatch. A demonstration video is available at: https://youtu.be/bN8ZowZKPMI.
Paper Structure (16 sections, 6 equations, 7 figures, 7 tables)

This paper contains 16 sections, 6 equations, 7 figures, 7 tables.

Table of Contents

  1. INTRODUCTION
  2. METHOD
  3. Task Definition and Stages
  4. System Setup
  5. Data Collection
  6. Data Analysis
  7. Policy Adaptation
  8. Deployment
  9. EXPERIMENTS
  10. Experimental Setup
  11. Metrics
  12. Main Results
  13. Ablations
  14. Comparisons
  15. Discussion
  16. ...and 1 more sections

Figures (7)

  • Figure 1: HarvestFlex end-to-end closed-loop harvesting pipeline: three-view RGB observations and robot state, conditioned on a language goal, were mapped by a VLA policy to 8-D actions for real-world execution.
  • Figure 2: Real-world strawberry picking scene and camera configuration.
  • Figure 3: Example three-view observations in our dataset: (a) left scene camera (D455), (b) wrist camera (D405), and (c) right scene camera (D455).
  • Figure 4: VR teleoperation interface and single-operator control scheme. (a) Schematic of the third-person viewpoint used to illustrate the teleoperation setup. (b) First-person view from the operator's headset recorded during demonstration collection.
  • Figure 5: VR controller button map for one-person operation.
  • ...and 2 more figures