Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Autonomous Manipulator Control with Human-in-loop for Uncertain Assembly Environments

Ashutosh Mishra, Shreya Santra, Hazal Gozbasi, Kentaro Uno, Kazuya Yoshida

TL;DR

This work tackles autonomous robotic deployment of flexible solar panels in uncertain lunar-like environments by integrating Human-in-the-Loop (HITL) control with real-time sensing and a digital twin. The system combines a UR16e manipulator, an extendable ladder, ArUco-based vision, RGB-D pose estimation, and force-torque sensing, enabling dynamic payload updates and adaptive control during deployment. Key innovations include a coarse-to-fine vision thresholding scheme, RRT-based motion planning, and a single-axis Sequential Motion Planning (SMP) strategy that mitigates sand sinkage, with HITL providing intervention during ambiguous phases. Field tests at JAXA validate the approach under lunar-like lighting and sandy terrain, confirming improved stability and reliability though future work is needed to address reduced gravity, vacuum, and extreme thermal conditions for on-moon operations.

Abstract

This study presents an advanced approach to enhance robotic manipulation in uncertain and challenging environments, with a focus on autonomous operations augmented by human-in-the-loop (HITL) control for lunar missions. By integrating human decision-making with autonomous robotic functions, the research improves task reliability and efficiency for space applications. The key task addressed is the autonomous deployment of flexible solar panels using an extendable ladder-like structure and a robotic manipulator with real-time feedback for precision. The manipulator relays position and force-torque data, enabling dynamic error detection and adaptive control during deployment. To mitigate the effects of sinkage, variable payload, and low-lighting conditions, efficient motion planning strategies are employed, supplemented by human control that allows operators to intervene in ambiguous scenarios. Digital twin simulation enhances system robustness by enabling continuous feedback, iterative task refinement, and seamless integration with the deployment pipeline. The system has been tested to validate its performance in simulated lunar conditions and ensure reliability in extreme lighting, variable terrain, changing payloads, and sensor limitations.

Enhancing Autonomous Manipulator Control with Human-in-loop for Uncertain Assembly Environments

TL;DR

This work tackles autonomous robotic deployment of flexible solar panels in uncertain lunar-like environments by integrating Human-in-the-Loop (HITL) control with real-time sensing and a digital twin. The system combines a UR16e manipulator, an extendable ladder, ArUco-based vision, RGB-D pose estimation, and force-torque sensing, enabling dynamic payload updates and adaptive control during deployment. Key innovations include a coarse-to-fine vision thresholding scheme, RRT-based motion planning, and a single-axis Sequential Motion Planning (SMP) strategy that mitigates sand sinkage, with HITL providing intervention during ambiguous phases. Field tests at JAXA validate the approach under lunar-like lighting and sandy terrain, confirming improved stability and reliability though future work is needed to address reduced gravity, vacuum, and extreme thermal conditions for on-moon operations.

Abstract

This study presents an advanced approach to enhance robotic manipulation in uncertain and challenging environments, with a focus on autonomous operations augmented by human-in-the-loop (HITL) control for lunar missions. By integrating human decision-making with autonomous robotic functions, the research improves task reliability and efficiency for space applications. The key task addressed is the autonomous deployment of flexible solar panels using an extendable ladder-like structure and a robotic manipulator with real-time feedback for precision. The manipulator relays position and force-torque data, enabling dynamic error detection and adaptive control during deployment. To mitigate the effects of sinkage, variable payload, and low-lighting conditions, efficient motion planning strategies are employed, supplemented by human control that allows operators to intervene in ambiguous scenarios. Digital twin simulation enhances system robustness by enabling continuous feedback, iterative task refinement, and seamless integration with the deployment pipeline. The system has been tested to validate its performance in simulated lunar conditions and ensure reliability in extreme lighting, variable terrain, changing payloads, and sensor limitations.

Paper Structure

This paper contains 17 sections, 8 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. System Overview
  5. Hardware Configuration
  6. Software and Control Framework
  7. Vision-feedback and Pose Estimation
  8. Motion Planning and Collision Avoidance
  9. Dynamic Payload Updates and Human-in-the-Loop
  10. Simulated Digital Twin and Continuous Integration
  11. Experiment Procedure
  12. Results and Analysis
  13. Validation and in-lab Testing
  14. Field Testing at JAXA
  15. Mitigating Sand Sinkage through Single-Axis Sequential Motion Planning
  16. ...and 2 more sections

Figures (7)

  • Figure 1: Setup for the solar panel deployment at the experiment site with low-illumination and loose sinking soil
  • Figure 2: Rviz visualization and Digital twin simulations in IsaacSim
  • Figure 3: Continuous Integration and Operator GUI
  • Figure 4: Experiment Procedure Flow Chart
  • Figure 5: Results of in-lab testing of Autonomous Deployment
  • ...and 2 more figures