CACTO-BIC: Scalable Actor-Critic Learning via Biased Sampling and GPU-Accelerated Trajectory Optimization

TL;DR

CACTO-BIC tackles the scalability gap in hybrid trajectory optimization and reinforcement learning by biasing initial-state sampling around informative regions indicated by discontinuities in the locally optimal value function and by using a learned uncertainty model to identify these regions. It further accelerates the workflow with a GPU-based implementation that runs TO and neural-network training on accelerators, achieving substantial speedups and enabling real-time applicability on high-dimensional systems like AlienGO. Empirically, CACTO-BIC improves sample efficiency by roughly 2.5–3.5× and reduces wall-clock time versus CACTO and PPO, while maintaining comparable solution quality across benchmarks and hardware experiments. The work demonstrates that biased initialization, together with GPU acceleration, can scale TO–RL hybrids to complex robotic control tasks with practical real-time performance.

Abstract

Trajectory Optimization (TO) and Reinforcement Learning (RL) offer complementary strengths for solving optimal control problems. TO efficiently computes locally optimal solutions but can struggle with non-convexity, while RL is more robust to non-convexity at the cost of significantly higher computational demands. CACTO (Continuous Actor-Critic with Trajectory Optimization) was introduced to combine these advantages by learning a warm-start policy that guides the TO solver towards low-cost trajectories. However, scalability remains a key limitation, as increasing system complexity significantly raises the computational cost of TO. This work introduces CACTO-BIC to address these challenges. CACTO-BIC improves data efficiency by biasing initial-state sampling leveraging a property of the value function associated with locally optimal policies; moreover, it reduces computation time by exploiting GPU acceleration. Empirical evaluations show improved sample efficiency and faster computation compared to CACTO. Comparisons with PPO demonstrate that our approach can achieve similar solutions in less time. Finally, experiments on the AlienGO quadruped robot demonstrate that CACTO-BIC can scale to high-dimensional systems and is suitable for real-time applications.

Figures (8)

• Figure 1: Cost and Value obtained with TO using a naive initial guess. The critic smooths the Value's discontinuities.
• Figure 2: Overview of CACTO-BIC with biased initial-state sampling.
• Figure 3: Cost function excluding the control effort term, with target set at $[-7,0]$.
• Figure 4: Median (across 5 runs) of the mean cost (across initial conditions) starting from the Hard Region for the point mass (top), the Dubins car, and the manipulator (bottom). Shaded areas represent first and third quartiles. Data are sampled every n updates. When multiple measurements occur at the same TO episode count, the last one is plotted.
• Figure 5: Speedups for TO on GPU (w.r.t. CPU) for the point mass, the Dubins car, and the manipulator. Dashed lines represent speedups with random initial states and maximum number of iterations, selected as described in Section \ref{['ssec:iteration_number']}, while solid lines represent speedups with initial conditions selected using CACTO-BIC and maximum number of iterations set as in the two versions.