Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Contact-Driven Framework for Manipulating in the Blind

Muhammad Suhail Saleem, Lai Yuan, Maxim Likhachev

TL;DR

This work presents a complete, contact-driven framework for manipulation in visually occluded environments by tightly coupling torque-based contact detection/localization, occupancy estimation with learned priors, and planning that accounts for uncertainty. It integrates a momentum-observer-based contact detector with a contact particle filter, two occupancy-prediction models (CNN and diffusion), and planning via CHS and CMAX to maintain completeness while exploiting structural priors. Across simulated and real-world tasks—valve manipulation under a sink and object retrieval from cluttered shelves—the approach achieves up to 2× faster task completion than baselines and shows robustness to localization and occupancy prediction noise. The results demonstrate that combining contact signals with learned workspace priors enables robust, efficient manipulation in unknown environments, with practical impact for service robots operating in cluttered or occluded spaces.

Abstract

Robots often face manipulation tasks in environments where vision is inadequate due to clutter, occlusions, or poor lighting--for example, reaching a shutoff valve at the back of a sink cabinet or locating a light switch above a crowded shelf. In such settings, robots, much like humans, must rely on contact feedback to distinguish free from occupied space and navigate around obstacles. Many of these environments often exhibit strong structural priors--for instance, pipes often span across sink cabinets--that can be exploited to anticipate unseen structure and avoid unnecessary collisions. We present a theoretically complete and empirically efficient framework for manipulation in the blind that integrates contact feedback with structural priors to enable robust operation in unknown environments. The framework comprises three tightly coupled components: (i) a contact detection and localization module that utilizes joint torque sensing with a contact particle filter to detect and localize contacts, (ii) an occupancy estimation module that uses the history of contact observations to build a partial occupancy map of the workspace and extrapolate it into unexplored regions with learned predictors, and (iii) a planning module that accounts for the fact that contact localization estimates and occupancy predictions can be noisy, computing paths that avoid collisions and complete tasks efficiently without eliminating feasible solutions. We evaluate the system in simulation and in the real world on a UR10e manipulator across two domestic tasks--(i) manipulating a valve under a kitchen sink surrounded by pipes and (ii) retrieving a target object from a cluttered shelf. Results show that the framework reliably solves these tasks, achieving up to a 2x reduction in task completion time compared to baselines, with ablations confirming the contribution of each module.

A Contact-Driven Framework for Manipulating in the Blind

TL;DR

This work presents a complete, contact-driven framework for manipulation in visually occluded environments by tightly coupling torque-based contact detection/localization, occupancy estimation with learned priors, and planning that accounts for uncertainty. It integrates a momentum-observer-based contact detector with a contact particle filter, two occupancy-prediction models (CNN and diffusion), and planning via CHS and CMAX to maintain completeness while exploiting structural priors. Across simulated and real-world tasks—valve manipulation under a sink and object retrieval from cluttered shelves—the approach achieves up to 2× faster task completion than baselines and shows robustness to localization and occupancy prediction noise. The results demonstrate that combining contact signals with learned workspace priors enables robust, efficient manipulation in unknown environments, with practical impact for service robots operating in cluttered or occluded spaces.

Abstract

Robots often face manipulation tasks in environments where vision is inadequate due to clutter, occlusions, or poor lighting--for example, reaching a shutoff valve at the back of a sink cabinet or locating a light switch above a crowded shelf. In such settings, robots, much like humans, must rely on contact feedback to distinguish free from occupied space and navigate around obstacles. Many of these environments often exhibit strong structural priors--for instance, pipes often span across sink cabinets--that can be exploited to anticipate unseen structure and avoid unnecessary collisions. We present a theoretically complete and empirically efficient framework for manipulation in the blind that integrates contact feedback with structural priors to enable robust operation in unknown environments. The framework comprises three tightly coupled components: (i) a contact detection and localization module that utilizes joint torque sensing with a contact particle filter to detect and localize contacts, (ii) an occupancy estimation module that uses the history of contact observations to build a partial occupancy map of the workspace and extrapolate it into unexplored regions with learned predictors, and (iii) a planning module that accounts for the fact that contact localization estimates and occupancy predictions can be noisy, computing paths that avoid collisions and complete tasks efficiently without eliminating feasible solutions. We evaluate the system in simulation and in the real world on a UR10e manipulator across two domestic tasks--(i) manipulating a valve under a kitchen sink surrounded by pipes and (ii) retrieving a target object from a cluttered shelf. Results show that the framework reliably solves these tasks, achieving up to a 2x reduction in task completion time compared to baselines, with ablations confirming the contribution of each module.
Paper Structure (18 sections, 9 equations, 3 figures, 2 tables)

This paper contains 18 sections, 9 equations, 3 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Problem Setup
  4. METHODOLOGY
  5. Contact Detection and Localization Module
  6. External Momentum Observer for Detection
  7. Contact Particle Filter for Localization
  8. Robustifying Contact Localization
  9. Occupancy Estimation
  10. CNN Predictor
  11. Diffusion Predictor
  12. Planning Module
  13. Integration with CHS
  14. Integration with CMAX
  15. RESULTS
  16. ...and 3 more sections

Figures (3)

  • Figure 1: A robot reaching a shutoff valve at the back of a cluttered cabinet. As visual sensing is occluded by the shelf door (top), the robot relies on contact feedback to navigate around obstacles.
  • Figure 2: The collision hypothesis set $\kappa_i$ (in green) corresponds to the voxels intersecting with the active surface of the robot at the time of collision.
  • Figure 3: Example runs of the framework on the object retrieval task (top) and valve manipulation task (bottom). In both cases, the robot initially collides with environmental obstacles (highlighted in red) before identifying a feasible path that allows it to successfully reach the target.