Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Task-Relevant and Irrelevant Region-Aware Augmentation for Generalizable Vision-Based Imitation Learning in Agricultural Manipulation

Shun Hattori, Hikaru Sasaki, Takumi Hachimine, Yusuke Mizutani, Takamitsu Matsubara

TL;DR

DRAIL is evaluated on diffusion policy-based visuomotor controllers through robot experiments on artificial vegetable harvesting and real lettuce defective leaf picking preparation tasks, showing consistent improvements in success rates under unseen visual conditions compared to baseline methods.

Abstract

Vision-based imitation learning has shown promise for robotic manipulation; however, its generalization remains limited in practical agricultural tasks. This limitation stems from scarce demonstration data and substantial visual domain gaps caused by i) crop-specific appearance diversity and ii) background variations. To address this limitation, we propose Dual-Region Augmentation for Imitation Learning (DRAIL), a region-aware augmentation framework designed for generalizable vision-based imitation learning in agricultural manipulation. DRAIL explicitly separates visual observations into task-relevant and task-irrelevant regions. The task-relevant region is augmented in a domain-knowledge-driven manner to preserve essential visual characteristics, while the task-irrelevant region is aggressively randomized to suppress spurious background correlations. By jointly handling both sources of visual variation, DRAIL promotes learning policies that rely on task-essential features rather than incidental visual cues. We evaluate DRAIL on diffusion policy-based visuomotor controllers through robot experiments on artificial vegetable harvesting and real lettuce defective leaf picking preparation tasks. The results show consistent improvements in success rates under unseen visual conditions compared to baseline methods. Further attention analysis and representation generalization metrics indicate that the learned policies rely more on task-essential visual features, resulting in enhanced robustness and generalization.

Task-Relevant and Irrelevant Region-Aware Augmentation for Generalizable Vision-Based Imitation Learning in Agricultural Manipulation

TL;DR

DRAIL is evaluated on diffusion policy-based visuomotor controllers through robot experiments on artificial vegetable harvesting and real lettuce defective leaf picking preparation tasks, showing consistent improvements in success rates under unseen visual conditions compared to baseline methods.

Abstract

Vision-based imitation learning has shown promise for robotic manipulation; however, its generalization remains limited in practical agricultural tasks. This limitation stems from scarce demonstration data and substantial visual domain gaps caused by i) crop-specific appearance diversity and ii) background variations. To address this limitation, we propose Dual-Region Augmentation for Imitation Learning (DRAIL), a region-aware augmentation framework designed for generalizable vision-based imitation learning in agricultural manipulation. DRAIL explicitly separates visual observations into task-relevant and task-irrelevant regions. The task-relevant region is augmented in a domain-knowledge-driven manner to preserve essential visual characteristics, while the task-irrelevant region is aggressively randomized to suppress spurious background correlations. By jointly handling both sources of visual variation, DRAIL promotes learning policies that rely on task-essential features rather than incidental visual cues. We evaluate DRAIL on diffusion policy-based visuomotor controllers through robot experiments on artificial vegetable harvesting and real lettuce defective leaf picking preparation tasks. The results show consistent improvements in success rates under unseen visual conditions compared to baseline methods. Further attention analysis and representation generalization metrics indicate that the learned policies rely more on task-essential visual features, resulting in enhanced robustness and generalization.
Paper Structure (21 sections, 5 equations, 50 figures, 5 tables)

This paper contains 21 sections, 5 equations, 50 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Automation of Agricultural Tasks with Imitation Learning
  4. Data Augmentation in Imitation Learning
  5. Dual-Region Augmentation for Imitation Learning
  6. Imitation Learning with Data Augmentation
  7. Visual Observation Augmentation
  8. Task-Relevant Region Extraction
  9. Task-Relevant Region Augmentation
  10. Task-Irrelevant Region Augmentation
  11. Experiment
  12. General Settings
  13. Representation Generalization Metrics
  14. Experiments with Artificial Vegetable Harvesting Tasks
  15. Settings
  16. ...and 6 more sections

Figures (50)

  • Figure 1: Robot automation system
  • Figure 2: Attention regions of visuomotor policy
  • Figure 4: Overview of Dual-Region Augmentation for Imitation Learning (DRAIL). (a) shows task-relevant segmentation. A mask of the task-relevant region is initialized on the first frame of visual observations using SAM and then propagated to subsequent frames using XMem++, yielding per-frame masks. (b) shows task-relevant augmentation. The task-relevant region undergoes task-specific augmentation. (c) shows task-irrelevant randomization. The task-irrelevant region is perturbed using PixMix. The two streams are composited to form augmented data, repeated across all demonstration episodes.
  • Figure 5: Robot system
  • Figure 6: Robot gripper
  • ...and 45 more figures