Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HyReach: Vision-Guided Hybrid Manipulator Reaching in Unseen Cluttered Environments

Shivani Kamtikar, Kendall Koe, Justin Wasserman, Samhita Marri, Benjamin Walt, Naveen Kumar Uppalapati, Girish Krishnan, Girish Chowdhary

Abstract

As robotic systems increasingly operate in unstructured, cluttered, and previously unseen environments, there is a growing need for manipulators that combine compliance, adaptability, and precise control. This work presents a real-time hybrid rigid-soft continuum manipulator system designed for robust open-world object reaching in such challenging environments. The system integrates vision-based perception and 3D scene reconstruction with shape-aware motion planning to generate safe trajectories. A learning-based controller drives the hybrid arm to arbitrary target poses, leveraging the flexibility of the soft segment while maintaining the precision of the rigid segment. The system operates without environment-specific retraining, enabling direct generalization to new scenes. Extensive real-world experiments demonstrate consistent reaching performance with errors below 2 cm across diverse cluttered setups, highlighting the potential of hybrid manipulators for adaptive and reliable operation in unstructured environments.

HyReach: Vision-Guided Hybrid Manipulator Reaching in Unseen Cluttered Environments

Abstract

As robotic systems increasingly operate in unstructured, cluttered, and previously unseen environments, there is a growing need for manipulators that combine compliance, adaptability, and precise control. This work presents a real-time hybrid rigid-soft continuum manipulator system designed for robust open-world object reaching in such challenging environments. The system integrates vision-based perception and 3D scene reconstruction with shape-aware motion planning to generate safe trajectories. A learning-based controller drives the hybrid arm to arbitrary target poses, leveraging the flexibility of the soft segment while maintaining the precision of the rigid segment. The system operates without environment-specific retraining, enabling direct generalization to new scenes. Extensive real-world experiments demonstrate consistent reaching performance with errors below 2 cm across diverse cluttered setups, highlighting the potential of hybrid manipulators for adaptive and reliable operation in unstructured environments.
Paper Structure (5 sections, 1 equation, 2 figures)

This paper contains 5 sections, 1 equation, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Perception
  5. Path Planning with Shape Estimation

Figures (2)

  • Figure 1: Our system solves reaching tasks while using a hybrid rigid-soft continuum arm system. Setup consists of a B3 (three bending actuators) soft continuum arm with a small RGB camera mounted on a 6DOF rigid manipulator. The setup also has a magnetic sensor (used only for data collection) that measures the pose of the end effector. We show two overlayed snapshots of the manipulator reaching toward a goal object through a cluttered environment.
  • Figure 2: Our pipeline for real-time reaching and control of a hybrid manipulator in complex, unstructured environments. The pipeline comprises goal detection, 3D reconstruction, shape-informed path planning, and a learned controller for hybrid manipulators. 3D reconstruction, integrated with an occupancy grid, enhances scene understanding and identifies traversable areas. Shape-informed path planning optimizes paths by effectively navigating around obstacles. Additionally, our hybrid manipulator controller enables actuation to any arbitrary pose within the workspace.