Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Autonomous loading of ore piles with Load-Haul-Dump machines using Deep Reinforcement Learning

Rodrigo Salas, Francisco Leiva, Javier Ruiz-del-Solar

TL;DR

This work presents a deep reinforcement learning-based approach to train controllers for the autonomous loading of ore piles with a Load-Haul-Dump (LHD) machine, and shows good performance in the real world, achieving fill factors between 71-94%, and less wheel drift than the other baselines during the loading maneuvers.

Abstract

This work presents a deep reinforcement learning-based approach to train controllers for the autonomous loading of ore piles with a Load-Haul-Dump (LHD) machine. These controllers must perform a complete loading maneuver, filling the LHD's bucket with material while avoiding wheel drift, dumping material, or getting stuck in the pile. The training process is conducted entirely in simulation, using a simple environment that leverages the Fundamental Equation of Earth-Moving Mechanics so as to achieve a low computational cost. Two different types of policies are trained: one with a hybrid action space and another with a continuous action space. The RL-based policies are evaluated both in simulation and in the real world using a scaled LHD and a scaled muck pile, and their performance is compared to that of a heuristics-based controller and human teleoperation. Additional real-world experiments are performed to assess the robustness of the RL-based policies to measurement errors in the characterization of the piles. Overall, the RL-based controllers show good performance in the real world, achieving fill factors between 71-94%, and less wheel drift than the other baselines during the loading maneuvers. A video showing the training environment and the learned behavior in simulation, as well as some of the performed experiments in the real world, can be found in https://youtu.be/jOpA1rkwhDY.

Autonomous loading of ore piles with Load-Haul-Dump machines using Deep Reinforcement Learning

TL;DR

This work presents a deep reinforcement learning-based approach to train controllers for the autonomous loading of ore piles with a Load-Haul-Dump (LHD) machine, and shows good performance in the real world, achieving fill factors between 71-94%, and less wheel drift than the other baselines during the loading maneuvers.

Abstract

This work presents a deep reinforcement learning-based approach to train controllers for the autonomous loading of ore piles with a Load-Haul-Dump (LHD) machine. These controllers must perform a complete loading maneuver, filling the LHD's bucket with material while avoiding wheel drift, dumping material, or getting stuck in the pile. The training process is conducted entirely in simulation, using a simple environment that leverages the Fundamental Equation of Earth-Moving Mechanics so as to achieve a low computational cost. Two different types of policies are trained: one with a hybrid action space and another with a continuous action space. The RL-based policies are evaluated both in simulation and in the real world using a scaled LHD and a scaled muck pile, and their performance is compared to that of a heuristics-based controller and human teleoperation. Additional real-world experiments are performed to assess the robustness of the RL-based policies to measurement errors in the characterization of the piles. Overall, the RL-based controllers show good performance in the real world, achieving fill factors between 71-94%, and less wheel drift than the other baselines during the loading maneuvers. A video showing the training environment and the learned behavior in simulation, as well as some of the performed experiments in the real world, can be found in https://youtu.be/jOpA1rkwhDY.
Paper Structure (28 sections, 16 equations, 19 figures, 6 tables)

This paper contains 28 sections, 16 equations, 19 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Proposed approach
  4. Problem formulation
  5. Simulation environment
  6. The loading task as a POMDP
  7. Episodic settings
  8. Observations and actions
  9. Reward function
  10. Shovel trajectory reward $(r_\text{\normalfont traj}^t)$
  11. Mid goal reward $(r_\text{\normalfont midgoal}^t)$
  12. Inactivity reward $(r_\text{\normalfont inact}^t)$
  13. Drift reward $(r_\text{\normalfont drift}^t)$
  14. Stuck reward $(r_\text{\normalfont stuck}^t)$
  15. Dump reward $(r_{\text{\normalfont dump}}^t)$
  16. ...and 13 more sections

Figures (19)

  • Figure 1: Diagram of the side view of a sublevel stoping draw point from which an LHD has to load material (adapted from tampier2021autonomous, with permission of the authors).
  • Figure 2: Diagram of the muck pile division into voxels. Each column of voxels (one of them marked in gray) has a width $w$.
  • Figure 3: Illustration to represent the side view of the muck pile, and the ideal volume of material that would be loaded if the bucket tip would follow a certain trajectory.
  • Figure 4: Division of the muck pile profile into zones in the XZ plane. The blue zone corresponds to the "end zone", the yellow zone to the "permitted zone", and the red zone to the "restricted zone". All these zones are referenced to the $\{\text{M}\}$ frame.
  • Figure 5: Diagram of the LHD's arm joint angles and their references, $\phi_{\text{boom}}$ and $\phi_{\text{bucket}}$, and the bucket tip (shovel) position $p_{\text{shovel}} = (x_{\text{shovel}}, y_{\text{shovel}}, z_{\text{shovel}})$ and pitch angle $\phi_{\text{shovel}}$.
  • ...and 14 more figures