Agent-Driven Autonomous Reinforcement Learning Research: Iterative Policy Improvement for Quadruped Locomotion

Abstract

This paper documents a case study in agent-driven autonomous reinforcement learning research for quadruped locomotion. The setting was not a fully self-starting research system. A human provided high-level directives through an agentic coding environment, while an agent carried out most of the execution loop: reading code, diagnosing failures, editing reward and terrain configurations, launching and monitoring jobs, analyzing intermediate metrics, and proposing the next wave of experiments. Across more than 70 experiments organized into fourteen waves on a DHAV1 12-DoF quadruped in Isaac Lab, the agent progressed from early rough-terrain runs with mean reward around 7 to a best logged Wave 12 run, exp063, with velocity error 0.263 and 97\% timeout over 2000 iterations, independently reproduced five times across different GPUs. The archive also records several concrete autonomous research decisions: isolating PhysX deadlocks to terrain sets containing boxes and stair-like primitives, porting four reward terms from openly available reference implementations \cite{deeprobotics, rlsar}, correcting Isaac Sim import and bootstrapping issues, reducing environment count for diagnosis, terminating hung runs, and pivoting effort away from HIM after repeated terrain=0.0 outcomes. Relative to the AutoResearch paradigm \cite{autoresearch}, this case study operates in a more failure-prone robotics RL setting with multi-GPU experiment management and simulator-specific engineering constraints. The contribution is empirical and documentary: it shows that an agent can materially execute the iterative RL research loop in this domain with limited human intervention, while also making clear where human direction still shaped the agenda.

Figures (9)

• Figure 1: Mean reward vs. iteration for key experiments across waves. Later waves with refined reward weights and terrain configuration show consistently higher and faster-rising reward curves.
• Figure 2: Planar velocity tracking error across key experiments. Lower values indicate better command following. The best run (exp063) converges to error 0.263, reproduced five times.
• Figure 3: Experiment completion rate by wave. Red: $<$50%, yellow: 50--80%, green: $>$80%. Fractions show completed/launched. The PhysX hang problem persisted throughout the study, with Wave 11 achieving the first 100% completion rate.
• Figure 4: HIM vs Basic PPO on DR terrain with identical reward configuration. (a) Mean reward, (b) velocity tracking error, (c) episode survival rate. Basic PPO dramatically outperforms HIM across all metrics in this specific project configuration.
• Figure 5: Best velocity error per wave. The optimization converges at Wave 12 (vel_err=0.263). Waves 13--14 explored the remaining parameter space without improvement. Vertical gray lines indicate waves without comparable velocity error data (early diagnostic or HIM-only waves).