Large Scale Robotic Material Handling: Learning, Planning, and Control

TL;DR

This work presents a comprehensive framework for the autonomous execution of large-scale material handling tasks on a full scale, and is believed to be the first complete automation of material handling tasks on a full scale.

Abstract

Bulk material handling involves the efficient and precise moving of large quantities of materials, a core operation in many industries, including cargo ship unloading, waste sorting, construction, and demolition. These repetitive, labor-intensive, and safety-critical operations are typically performed using large hydraulic material handlers equipped with underactuated grippers. In this work, we present a comprehensive framework for the autonomous execution of large-scale material handling tasks. The system integrates specialized modules for environment perception, pile attack point selection, path planning, and motion control. The main contributions of this work are two reinforcement learning-based modules: an attack point planner that selects optimal grasping locations on the material pile to maximize removal efficiency and minimize the number of scoops, and a robust trajectory following controller that addresses the precision and safety challenges associated with underactuated grippers in movement, while utilizing their free-swinging nature to release material through dynamic throwing. We validate our framework through real-world experiments on a 40 t material handler in a representative worksite, focusing on two key tasks: high-throughput bulk pile management and high-precision truck loading. Comparative evaluations against human operators demonstrate the system's effectiveness in terms of precision, repeatability, and operational safety. To the best of our knowledge, this is the first complete automation of material handling tasks on a full scale.

Figures (21)

• Figure 1: The proposed framework performing the dump truck loading routine. The framework consists of key modules such as the RL attack point planner, the obstacle-avoiding path planner, the RL waypoint-following controller, and the RL throwing controller. This figure demonstrates the orchestration of these modules in a single grasp-and-dump cycle. First, the RL attack point planner proposes the optimal attack point based on LiDAR observation of the pile geometry, shown as light blue dots (Subfigures A and B). Then, the sampling-based path planner proposes a collision-free path, shown as a green dotted line. A sequence of waypoints sampled from the path, represented by red spheres, is tracked by the RL throwing policy (Subfigures C and D). Finally, the RL throwing controller dumps the load at the user-defined location in the the truck bed (Subfigures E, F). In addition to proprioceptive data, the framework uses the LiDAR pointclouds for exteroceptive observations: the cropped pointcloud of the pile is directly used to grasp planning (Subfigure B), while voxels converted from the pointclouds are used to represent obstacles in the environment (Subfigure D, F).
• Figure 2: The material handler used in this work has an operational range of about 20m and weighs around 40t. It is equipped with a 1.5t grabshell gripper designed for bulk material, with a maximum load capacity of 1.5 m. When fully open, it covers an area of approximately $2.0\times1.5 m \squared$.
• Figure 3: The LUTs are constructed from 37 datapoints each, obtained by measuring the steady-state velocity response of each joint under step excitations. Each value is computed by averaging the final $2s$ of recorded velocities for a given input signal, in line with the steady-state assumption. While the boom behavior is relatively symmetric, the stick dynamics exhibit a pronounced asymmetry in the input-velocity relationship.
• Figure 4: Component diagram of our framework. Given a specified grasping region and dumping target, the pipeline autonomously controls the machine in closed loop. The planning module takes into account the user input and LiDAR point cloud data to generate collision-free paths toward either the optimal grasp point or the designated dumping target. These paths are followed by high-level control policies that guide the end-effector through subsampled waypoints. A state machine manages the active controller, selecting between waypoint following, grasping, and throwing, ensuring that only one controller sends commands to the low-level control module at any time.
• Figure 5: Simulated soil for the RL attack point planner. Our flexible simulation allows for an easy adaptation to various geometries, such as an unconstrained pile on the ground (top) and bulk material in a container (bottom). The profiles shown are sampled at the beginning of each episode based on 2D Gaussian distributions.