Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompt-responsive Object Retrieval with Memory-augmented Student-Teacher Learning

Malte Mosbach, Sven Behnke

TL;DR

The paper addresses the challenge of enabling dexterous robotic manipulation guided by open-vocabulary prompts in cluttered scenes. It introduces a memory-augmented student-teacher framework in which a privileged teacher trained in simulation guides a perception-driven student that relies on SAM 2 detections and observation history, bridged by a memory module. The teacher is trained with PPO on privileged state and its behaviors are distilled via imitation within a DAgger framework; memory architectures (LSTM, Transformer, 1D-CNN) enable the student to implicitly infer the true object state from imperfect perception. Results show high success rates in both tabletop and cluttered-bin tasks and demonstrate transfer from simulation to real robots, highlighting the practical impact of combining open-vocabulary perception with memory-enabled policy learning for prompt-driven manipulation.

Abstract

Building models responsive to input prompts represents a transformative shift in machine learning. This paradigm holds significant potential for robotics problems, such as targeted manipulation amidst clutter. In this work, we present a novel approach to combine promptable foundation models with reinforcement learning (RL), enabling robots to perform dexterous manipulation tasks in a prompt-responsive manner. Existing methods struggle to link high-level commands with fine-grained dexterous control. We address this gap with a memory-augmented student-teacher learning framework. We use the Segment-Anything 2 (SAM 2) model as a perception backbone to infer an object of interest from user prompts. While detections are imperfect, their temporal sequence provides rich information for implicit state estimation by memory-augmented models. Our approach successfully learns prompt-responsive policies, demonstrated in picking objects from cluttered scenes. Videos and code are available at https://memory-student-teacher.github.io

Prompt-responsive Object Retrieval with Memory-augmented Student-Teacher Learning

TL;DR

The paper addresses the challenge of enabling dexterous robotic manipulation guided by open-vocabulary prompts in cluttered scenes. It introduces a memory-augmented student-teacher framework in which a privileged teacher trained in simulation guides a perception-driven student that relies on SAM 2 detections and observation history, bridged by a memory module. The teacher is trained with PPO on privileged state and its behaviors are distilled via imitation within a DAgger framework; memory architectures (LSTM, Transformer, 1D-CNN) enable the student to implicitly infer the true object state from imperfect perception. Results show high success rates in both tabletop and cluttered-bin tasks and demonstrate transfer from simulation to real robots, highlighting the practical impact of combining open-vocabulary perception with memory-enabled policy learning for prompt-driven manipulation.

Abstract

Building models responsive to input prompts represents a transformative shift in machine learning. This paradigm holds significant potential for robotics problems, such as targeted manipulation amidst clutter. In this work, we present a novel approach to combine promptable foundation models with reinforcement learning (RL), enabling robots to perform dexterous manipulation tasks in a prompt-responsive manner. Existing methods struggle to link high-level commands with fine-grained dexterous control. We address this gap with a memory-augmented student-teacher learning framework. We use the Segment-Anything 2 (SAM 2) model as a perception backbone to infer an object of interest from user prompts. While detections are imperfect, their temporal sequence provides rich information for implicit state estimation by memory-augmented models. Our approach successfully learns prompt-responsive policies, demonstrated in picking objects from cluttered scenes. Videos and code are available at https://memory-student-teacher.github.io
Paper Structure (18 sections, 3 figures, 5 tables)

This paper contains 18 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning-based Robotic Grasping
  4. Learning from Privileged Information
  5. Prompt-Guided Manipulation
  6. Method
  7. Overview
  8. Observations
  9. Learning from Privileged Information
  10. Memory-augmented Student-Teacher Learning
  11. Experimental Setup
  12. Environments
  13. Evaluation and Metrics
  14. Results
  15. Grasping from Tabletop Scenes
  16. ...and 3 more sections

Figures (3)

  • Figure 1: We present a student-teacher framework that enables learning interactive, prompt-responsive policies for object retrieval from cluttered scenes (Figure inspired by Shen2023).
  • Figure 2: We propose to train prompt-guided policies in two stages. First, the teacher policy is trained with model-free RL to solve the control problem from privileged information $\bm{s}_t^\mathrm{ext}$. Thereafter, the student policy is trained to imitate the teacher without access to $\bm{s}_t^\mathrm{ext}$, forcing it to implicitly infer the object state from the history of visual observations $\bm{o}_{[t-H, t]}^\mathrm{ext}$.
  • Figure 3: Training performance of (A) the teacher policies, (B) the student policies, and (C) the imitation loss over time steps. The dashed horizontal lines indicate the performance of the teacher policy used by DAgger.