Residual Reinforcement Learning for Waste-Container Lifting Using Large-Scale Cranes with Underactuated Tools

TL;DR

This work tackles high-precision container lifting with a truck-mounted hydraulic loader crane that features an underactuated discharge unit. It proposes residual reinforcement learning (RRL), which augments a stable nominal Cartesian controller with a learned residual policy, allowing improved trajectory tracking and swing suppression under unmodeled dynamics. The nominal controller comprises Cartesian admittance tracking, pendulum-aware anti-swing acceleration, and damped least-squares IK with a nullspace term, while a PPO-trained residual policy refines control during a critical alignment segment; training leverages episode initialization and domain randomization in Isaac Lab, with a final blending u=(1-λ)nor u + λ res u. In simulation, RRL yields higher lifting success rates, tighter tracking, and reduced swing than the nominal controller alone, with ablations showing complementary benefits of anti-swing and residual learning and robustness to parameter variations, pointing to promising real-world deployment after incorporating hydraulic dynamics and real hardware validation.

Abstract

This paper studies the container lifting phase of a waste-container recycling task in urban environments, performed by a hydraulic loader crane equipped with an underactuated discharge unit, and proposes a residual reinforcement learning (RRL) approach that combines a nominal Cartesian controller with a learned residual policy. All experiments are conducted in simulation, where the task is characterized by tight geometric tolerances between the discharge-unit hooks and the container rings relative to the overall crane scale, making precise trajectory tracking and swing suppression essential. The nominal controller uses admittance control for trajectory tracking and pendulum-aware swing damping, followed by damped least-squares inverse kinematics with a nullspace posture term to generate joint velocity commands. A PPO-trained residual policy in Isaac Lab compensates for unmodeled dynamics and parameter variations, improving precision and robustness without requiring end-to-end learning from scratch. We further employ randomized episode initialization and domain randomization over payload properties, actuator gains, and passive joint parameters to enhance generalization. Simulation results demonstrate improved tracking accuracy, reduced oscillations, and higher lifting success rates compared to the nominal controller alone.

Figures (7)

• Figure 1: Simulated waste-container handling scenario.
• Figure 2: Overall control architecture. The nominal controller outputs ${}^{\mathrm{nor}}u$ (TCP admittance + anti-swing + IK). A PPO-trained residual policy outputs ${}^{\mathrm{res}}u$. The final joint-velocity command is $u=(1-\lambda){}^{\mathrm{nor}}u+\lambda {}^{\mathrm{res}}u$ with error-dependent blending $\lambda$, and is applied to joint-level actuators.
• Figure 3: System overview. Left: crane kinematic model and task setup. Right: close-ups illustrating tight hook--ring tolerances.
• Figure 4: Randomized episode initialization: container spawned around the truck; TCP initialized in a cube above the container. TCP trajectory consists 3 segments (A, B, C) is generated according container and TCP position.
• Figure 5: Representative lifting episode: (a) reference vs. executed TCP trajectory; (b--g) motion sequence.