Visual Manipulation with Legs

TL;DR

A system that enables quadruped robots to interact with objects using their legs, inspired by non-prehensile manipulation is introduced, demonstrating more versatile object manipulation skills with legs than previous work.

Abstract

Animals use limbs for both locomotion and manipulation. We aim to equip quadruped robots with similar versatility. This work introduces a system that enables quadruped robots to interact with objects using their legs, inspired by non-prehensile manipulation. The system has two main components: a visual manipulation policy module and a loco-manipulator module. The visual manipulation policy, trained with reinforcement learning (RL) using point cloud observations and object-centric actions, decides how the leg should interact with the object. The loco-manipulator controller manages leg movements and body pose adjustments, based on impedance control and Model Predictive Control (MPC). Besides manipulating objects with a single leg, the system can select from the left or right leg based on critic maps and move objects to distant goals through base adjustment. Experiments evaluate the system on object pose alignment tasks in both simulation and the real world, demonstrating more versatile object manipulation skills with legs than previous work. Videos can be found at https://legged-manipulation.github.io/

Figures (12)

• Figure 1: Visual Manipulation with Legs. Our system operates in stages: 1) A depth camera captures the object's point cloud. 2) The point cloud and target pose are processed by our network. 3) The manipulation leg is chosen based on the highest Q-value. 4) Pre-contact, contact, and action details are sent to the low-level control system. 5) The control system uses impedance control to direct the selected leg, while a Model Predictive Controller maintains balance, sending torques to the robot.
• Figure 2: Control FSM. Our Finite State Machine (FSM) transition design follows a closed-loop approach, allowing for the repeated execution of manipulation actions through such a design.
• Figure 3: We employ RPM-Net for the registration process of a real robot, emphasizing successful registrations. In the illustration, the blue point cloud represents the source data captured by the camera, the yellow point cloud corresponds to the complete scan of the object, and the red point cloud shows the source data after transformation. Green lines indicate the flow vectors.
• Figure 4: We visualize the real robot trajectories. The semi-transparent overlies are the goal poses.
• Figure 5: Quantitative Results in Simulation and Generalization. We evaluate the performance of our method against flow, planning, and random location baselines in various object manipulation challenges. The left plot shows results for training objects across different manipulation tasks. The right plot demonstrates generalization to novel objects in a pushing task. Our method consistently outperforms the baselines across all tasks.