Improving CMA-ES Convergence Speed, Efficiency, and Reliability in Noisy Robot Optimization Problems

TL;DR

The paper tackles the problem of noisy, time-consuming experimental robot optimization by introducing Adaptive Sampling CMA-ES (AS-CMA), which allocates per-candidate evaluation time to maintain reliable sorting in CMA-ES. AS-CMA uses local landscape estimates and a target signal-to-noise ratio to dynamically adjust sampling time, enabling faster convergence and lower cumulative cost compared with CMA-ES with static sampling, Bayesian optimization, and KL-KG CMA-ES across four simulated landscapes and a real exoskeleton experiment. Across metrics of convergence reliability, time, and total cost, AS-CMA demonstrated consistent improvements, particularly in complex landscapes, while maintaining comparable setup complexity. The laboratory exoskeleton pilot validated the approach in a real-world setting, achieving substantial energy-cost reductions and corroborating the simulated results, with open-source code released for broader use.

Abstract

Experimental robot optimization often requires evaluating each candidate policy for seconds to minutes. The chosen evaluation time influences optimization because of a speed-accuracy tradeoff: shorter evaluations enable faster iteration, but are also more subject to noise. Here, we introduce a supplement to the CMA-ES optimization algorithm, named Adaptive Sampling CMA-ES (AS-CMA), which assigns sampling time to candidates based on predicted sorting difficulty, aiming to achieve consistent precision. We compared AS-CMA to CMA-ES and Bayesian optimization using a range of static sampling times in four simulated cost landscapes. AS-CMA converged on 98% of all runs without adjustment to its tunable parameter, and converged 24-65% faster and with 29-76% lower total cost than each landscape's best CMA-ES static sampling time. As compared to Bayesian optimization, AS-CMA converged more efficiently and reliably in complex landscapes, while in simpler landscapes, AS-CMA was less efficient but equally reliable. We deployed AS-CMA in an exoskeleton optimization experiment and found the optimizer's behavior was consistent with expectations. These results indicate that AS-CMA can improve optimization efficiency in the presence of noise while minimally affecting optimization setup complexity and tuning requirements.

Figures (16)

• Figure 1: Overview of sample allocation problem posed by noisy optimization and the proposed solution in adaptive sampling. If all candidates are measured for a medium amount of time (probability distribution for these measurements shown in green), excessive or inadequate sorting precision may result. If measurement time instead were based on the difference between the candidate and its nearest-costing neighbor $\delta$, an appropriate amount of sample time (probability distribution for these measurements shown in blue) could be allocated.
• Figure 2: Summary of how AS-CMA determines sample time. For each candidate $i$, the estimated cost difference to its nearest neighbor $\delta_{\text{nearest}}^{(i)}$ is unknown prior to sampling, so AS-CMA uses the nearest neighboring parameter's distance $d^{(i)}_{\text{nearest}}$ instead (Algorithm \ref{['alg: AS-CMA']}, line \ref{['line:nn_dist']}). Using $d^{(i)}_{\text{nearest}}$, the target noise level $\epsilon_i$ is found by using an approximation of the local cost landscape ($k_{avg}$ and $y_{avg}$) such that the sorting precision of the candidate and its nearest neighbor should achieve the desired precision $\beta$. (Algorithm \ref{['alg: AS-CMA']}, line \ref{['line:target_noise']}). Using $\epsilon_i$, sample time $t_i$ can be found by inverting noise model $\mathcal{E}(t)$ (Algorithm \ref{['alg: AS-CMA']}, line \ref{['line:meas_ti']}).
• Figure 3: Representation of numerical simulation framework. Given a set of candidate parameters, the simulation first determines the true cost (using a ground-truth cost landscape) and the desired sample time (using AS-CMA, which is shown, or a fixed sample time, not shown). Then, noise is introduced to the true cost based on the chosen sample time, producing the noisy cost. This noisy cost is returned to CMA-ES, which then generates new candidates to assess, and AS-CMA, for updating of $k_{avg}$ and $y_{avg}$.
• Figure 4: Average ($\pm1$ standard deviation) selected condition time by AS-CMA.
• Figure 5: Generation sorting accuracy (Spearman's $\rho$) over optimization runs. Triangles denote the average across all optimization time points for that landscape.