How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments

TL;DR

This paper addresses reproducibility concerns in deep reinforcement learning by clarifying how the number of random seeds affects statistical power when comparing two algorithms. It advocates two main approaches—the Welch's two-sample t-test and bootstrap confidence intervals—and provides a concrete workflow: run a pilot study to estimate variances, perform power calculations to choose sample sizes, and then apply the chosen tests while accounting for assumption deviations. The authors show that standard methods can miscalibrate error rates under small samples or non-normal data, recommending empirical checks, fewer multiple comparisons, and conservative sample-size planning. They also release code to implement the tests. Overall, the work offers practical, theory-grounded guidelines to design RL experiments with reliable statistical conclusions.

Abstract

Consistently checking the statistical significance of experimental results is one of the mandatory methodological steps to address the so-called "reproducibility crisis" in deep reinforcement learning. In this tutorial paper, we explain how the number of random seeds relates to the probabilities of statistical errors. For both the t-test and the bootstrap confidence interval test, we recall theoretical guidelines to determine the number of random seeds one should use to provide a statistically significant comparison of the performance of two algorithms. Finally, we discuss the influence of deviations from the assumptions usually made by statistical tests. We show that they can lead to inaccurate evaluations of statistical errors and provide guidelines to counter these negative effects. We make our code available to perform the tests.

Figures (8)

• Figure 1: $Algo1$ versus $Algo2$ are two famous Deep RL algorithms, here tested on the Half-Cheetah benchmark. The mean and confidence interval for 5 seeds are reported. We might consider that $Algo1$ outperforms $Algo2$ because there is not much overlap between the $95\%$ confidence intervals. But is it sufficient evidence that $Algo1$ really performs better? Below, we show that the performances of these algorithms is actually the same, and explain which methods should be used to have more reliable evidence of the (non-)difference among two algorithms.
• Figure 2: Representation of $H_0$ and $H_a$ under the $t$-test assumptions. Areas under the distributions represented in red, dark blue and light blue correspond to the probability of type-I error $\alpha$, type-II error $\beta$ and the statistical power $1-\beta$ respectively.
• Figure 3: DDPG with action perturbation versus DDPG with parameter perturbation tested in Half-Cheetah. Mean and $95\%$ confidence interval computed over $5$ seeds are reported. The figure shows a small difference in the empirical mean performances.
• Figure 4: Evolution of the probability of type-II error as a function of the sample size $N$ for various effect sizes $\epsilon$, when $(s_1,s_2)= (1341, 990)$ and $\alpha=0.05$. The requirement $0.2$ is represented by the horizontal dashed black line. The curve for $\epsilon=\overline{x}_{\textnormal{diff}}$ is represented by the semi-dashed black line.
• Figure 5: Performance of DDPG with action perturbation ($Algo1$) and parameter perturbation ($Algo2$) with $N=5$ seeds (left) and $N=10$ seeds (right). The $95\%$ confidence intervals on the right are smaller, because more evidence is available ($N$ larger). The underlying difference appears when $N$ grows.