Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Models

Qi Wu, Zipeng Fu, Xuxin Cheng, Xiaolong Wang, Chelsea Finn

TL;DR

This work presents a system for quadrupedal mobile manipulation in indoor environments that uses a front-mounted gripper for object manipulation, a low-level controller trained in simulation using egocentric depth for agile skills like climbing and whole-body tilting, and pre-trained vision-language models with third-person fisheye and an egocentric RGB camera for semantic understanding and command generation.

Abstract

Learning-based methods have achieved strong performance for quadrupedal locomotion. However, several challenges prevent quadrupeds from learning helpful indoor skills that require interaction with environments and humans: lack of end-effectors for manipulation, limited semantic understanding using only simulation data, and low traversability and reachability in indoor environments. We present a system for quadrupedal mobile manipulation in indoor environments. It uses a front-mounted gripper for object manipulation, a low-level controller trained in simulation using egocentric depth for agile skills like climbing and whole-body tilting, and pre-trained vision-language models (VLMs) with a third-person fisheye and an egocentric RGB camera for semantic understanding and command generation. We evaluate our system in two unseen environments without any real-world data collection or training. Our system can zero-shot generalize to these environments and complete tasks, like following user's commands to fetch a randomly placed stuff toy after climbing over a queen-sized bed, with a 60% success rate. Project website: https://helpful-doggybot.github.io/

Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Models

TL;DR

This work presents a system for quadrupedal mobile manipulation in indoor environments that uses a front-mounted gripper for object manipulation, a low-level controller trained in simulation using egocentric depth for agile skills like climbing and whole-body tilting, and pre-trained vision-language models with third-person fisheye and an egocentric RGB camera for semantic understanding and command generation.

Abstract

Learning-based methods have achieved strong performance for quadrupedal locomotion. However, several challenges prevent quadrupeds from learning helpful indoor skills that require interaction with environments and humans: lack of end-effectors for manipulation, limited semantic understanding using only simulation data, and low traversability and reachability in indoor environments. We present a system for quadrupedal mobile manipulation in indoor environments. It uses a front-mounted gripper for object manipulation, a low-level controller trained in simulation using egocentric depth for agile skills like climbing and whole-body tilting, and pre-trained vision-language models (VLMs) with a third-person fisheye and an egocentric RGB camera for semantic understanding and command generation. We evaluate our system in two unseen environments without any real-world data collection or training. Our system can zero-shot generalize to these environments and complete tasks, like following user's commands to fetch a randomly placed stuff toy after climbing over a queen-sized bed, with a 60% success rate. Project website: https://helpful-doggybot.github.io/
Paper Structure (21 sections, 3 equations, 3 figures, 5 tables)

This paper contains 21 sections, 3 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Hardware
  4. Learning a General Whole-Body Controller
  5. Phase 1: Training with Privileged Information
  6. Phase 2: Policy Distillation using Egocentric Depth
  7. Simulation Environments and Training Curricula
  8. Zero-Shot Deployment using VLMs
  9. Open-Vocabulary Detection, Segmentation and Tracking
  10. Navigation
  11. Grasping Objects
  12. Experiments
  13. Simulation Experiments
  14. Real-World Experiments
  15. Conclusion, Limitations & Future Directions
  16. ...and 6 more sections

Figures (3)

  • Figure 1: DoggyBot for Open-World Object Fetching. Using the coordination of commands from VLMs and a low-level whole-body policy, our robot can (A) climb up a bed to fetch a tennis ball, (B) bend down to pick up a water bottle, (C) climb up a couch to pick up a stuffed toy, and (D) climb down the bed after retrieving the tennis bell.
  • Figure 2: Hardware Setup. We use a Go2 quadruped and a custom-built 3D printed Gripper actuated by Dynamixel XM430-W350-T servo motor. An egocentric RealSense D435 is mounted on the top front of the robot with 30 degrees downwards.
  • Figure 3: System Overview. We use a two-phase framework to train a depth-based policy as the low-evel whole-body controller. During deployment, we use VLMs for open-vocabulary detection, segmentation and tracking models to provide velocity commands and pitch commands for the controller.