Efficient Model-Based Reinforcement Learning for Robot Control via Online Learning

TL;DR

This paper introduces an online model-based reinforcement learning framework for robot control that learns a dynamics model directly from real-world interaction and uses the model to generate approximate policy gradients for online policy updates. By coupling online model learning with a differentiable policy gradient computed via closed-loop Jacobians, and by enforcing strong regularization, the method achieves data-efficient learning with sublinear regret under standard online optimization assumptions. The theoretical analysis separates online policy and model-learning components to derive regret bounds that depend on gradient prediction error and distribution drift, respectively. Empirical evaluations on a hydraulic excavator (HEAP) and a cable-driven soft arm demonstrate rapid convergence and robust adaptation to changing payloads, outperforming some sim-to-real baselines in both speed and tracking accuracy, with only hours of real-world data required.

Abstract

We present an online model-based reinforcement learning algorithm suitable for controlling complex robotic systems directly in the real world. Unlike prevailing sim-to-real pipelines that rely on extensive offline simulation and model-free policy optimization, our method builds a dynamics model from real-time interaction data and performs policy updates guided by the learned dynamics model. This efficient model-based reinforcement learning scheme significantly reduces the number of samples to train control policies, enabling direct training on real-world rollout data. This significantly reduces the influence of bias in the simulated data, and facilitates the search for high-performance control policies. We adopt online learning analysis to derive sublinear regret bounds under standard stochastic online optimization assumptions, providing formal guarantees on performance improvement as more interaction data are collected. Experimental evaluations were performed on a hydraulic excavator arm and a soft robot arm, where the algorithm demonstrates strong sample efficiency compared to model-free reinforcement learning methods, reaching comparable performance within hours. Robust adaptation to shifting dynamics was also observed when the payload condition was randomized. Our approach paves the way toward efficient and reliable on-robot learning for a broad class of challenging control tasks.

Figures (12)

• Figure 1: A diagram showing the workflow of the proposed online model-based RL algorithm. It involves online model learning and policy optimization at the same time. The red blocks highlight the process of model learning, and the blue components show the workflow of policy optimization.
• Figure 2: HEAP, the autonomous excavator used in the experiments.
• Figure 3: Convergence of the online model-based RL algorithm in HEAP simulation. The plots show the average loss in model learning and control policy learning over 10 runs. The shaded area represents the standard deviation. Both mean and standard deviation are computed on a log scale.
• Figure 4: Evaluation of the control policy learned in simulation after 200 episodes. The plots show the tracking error of the control policy on a set of unseen reference trajectories that consists of straight lines and circles.
• Figure 5: Real-world deployment results on the HEAP excavator. The plots show convergence, evaluation metrics, and adaptation to payload changes.