Simulation-Aided Policy Tuning for Black-Box Robot Learning

TL;DR

This work tackles data-efficient robot learning under a black-box policy search setting by treating simulators as additional information sources. It introduces a derivative Gaussian process model and a local Bayesian optimization framework (HCI-GIBO) that guarantees high-probability policy improvements, and extends it to dual-information, sim-to-real scenarios (S-HCI-GIBO) with a SimToReal switching rule. The approach demonstrates superior data efficiency in synthetic high-dimensional benchmarks and validates performance gains on real robot tasks, including fine-tuning deep RL agents and full learning-from-scratch trajectory tracking. The results indicate practical impact for fast, reliable robot adaptation with limited hardware trials, while acknowledging limitations in local convergence and the need for informative priors; future work includes multi-simulator extensions and constrained optimization.

Abstract

How can robots learn and adapt to new tasks and situations with little data? Systematic exploration and simulation are crucial tools for efficient robot learning. We present a novel black-box policy search algorithm focused on data-efficient policy improvements. The algorithm learns directly on the robot and treats simulation as an additional information source to speed up the learning process. At the core of the algorithm, a probabilistic model learns the dependence of the policy parameters and the robot learning objective not only by performing experiments on the robot, but also by leveraging data from a simulator. This substantially reduces interaction time with the robot. Using this model, we can guarantee improvements with high probability for each policy update, thereby facilitating fast, goal-oriented learning. We evaluate our algorithm on simulated fine-tuning tasks and demonstrate the data-efficiency of the proposed dual-information source optimization algorithm. In a real robot learning experiment, we show fast and successful task learning on a robot manipulator with the aid of an imperfect simulator.

Figures (14)

• Figure 1: Experiment setup: A robot manipulator is balancing a planar pendulum and learning to follow the reference trajectory with the pendulum.
• Figure 2: Sequential black-box policy search: The search algorithm determines a query in order to gain more information on the performance function. The policy is then evaluated with the parameters given by the query. In each iteration, the search algorithm decides on its current guess of the best policy.
• Figure 3: Visualization of improvement confidence regions: Improvement confidence of gradient distribution $\nabla F_a$ and $\nabla F_b$ are 97% and 76%, respectively. Mean vectors are denoted by black arrows, and the contour lines show the density of the gradient distribution. If $L\eta = 1$ and true gradients are in regions filled with dots, gradient step using the mean vector improves the policy.
• Figure 4: Multi-fidelity queries: Comparison of uncertainties (blue shaded regions) after queries at same locations ($0.5$ and $0.7$ shown with star markers) from the simulator and the robot. Use toy objective function: $f$: blue line and $f_{\mathop{\mathrm{sim}}\limits}$: orange dash-dotted line; Posterior mean $\mu$ and $\nabla \mu$): blue dashed line; Current guess of the best policy: red cross marker.
• Figure 5: Visualization of the S-HCI-GIBO optimization process with a 1-dimensional function. Top: The objective functions of $f$ (blue line) and $f_{\mathop{\mathrm{sim}}\limits}$ (orange dash-dotted line). The red cross symbol refers to the current parameter $\theta_i$. The blue and orange star markers represent the samples that have been queried from $f$ and $f_{\mathop{\mathrm{sim}}\limits}$. The posterior mean $\mu( \theta, \mathop{\mathrm{IS}}\limits_{\mathop{\mathrm{real}}\limits})$ is shown with the blue dashed line. The shaded regions show the standard deviation. Middle: The posterior mean conditioned on star markers. Bottom: The acquisition function, i.e., the gradient information of the $f$ (blue line) and the $f_{\mathop{\mathrm{sim}}\limits}$ (orange dash-dotted line).

Theorems & Definitions (5)

• Lemma 1
• proof
• Theorem 1
• proof
• Remark 1