Benchmarking Reinforcement Learning via Stochastic Converse Optimality: Generating Systems with Known Optimal Policies

Abstract

The objective comparison of Reinforcement Learning (RL) algorithms is notoriously complex as outcomes and benchmarking of performances of different RL approaches are critically sensitive to environmental design, reward structures, and stochasticity inherent in both algorithmic learning and environmental dynamics. To manage this complexity, we introduce a rigorous benchmarking framework by extending converse optimality to discrete-time, control-affine, nonlinear systems with noise. Our framework provides necessary and sufficient conditions, under which a prescribed value function and policy are optimal for constructed systems, enabling the systematic generation of benchmark families via homotopy variations and randomized parameters. We validate it by automatically constructing diverse environments, demonstrating our framework's capacity for a controlled and comprehensive evaluation across algorithms. By assessing standard methods against a ground-truth optimum, our work delivers a reproducible foundation for precise and rigorous RL benchmarking.

Figures (7)

• Figure 1: Training dynamics snapshot: stage reward $r=-c$ over learning for representative algorithms on a fixed fixture under CRN evaluation. The core comparisons in this paper use absolute, oracle-referenced metrics (OptGap, regret).
• Figure 2: Converse-optimal benchmark pipeline: sampling, drift construction, validation, fixture export, paired evaluation, and reporting.
• Figure 3: Serial $n$-link planar arm (ConverseArm): absolute oracle-referenced performance (optimality gap and regret) across algorithms (rows) and control authority $p$ (columns), with separate panels for $n$.
• Figure 4: Compact result summaries. Macro-averaged optimality gaps and regrets across all experiments.
• Figure 5: Discounted reward bars ($r=-c$) with 95% CIs under CRN. Colors correspond to $p \in \{0.5, 0.6, 0.8\}.$