Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reproducibility and Statistical Methodology

Anthony Almudevar, Jacob Almudevar

Abstract

In 2015 the Open Science Collaboration (OSC) (Nosek et al 2015) published a highly influential paper which claimed that a large fraction of published results in the psychological sciences were not reproducible. In this article we review this claim from several points of view. We first offer an extended analysis of the methods used in that study. We show that the OSC methodology induces a bias that is able by itself to explain the discrepancy between the OSC estimates of reproducibility and other more optimistic estimates made by similar studies. The article also offers a more general literature review and discussion of reproducibility in experimental science. We argue, for both scientific and ethical reasons, that a considered balance of false positive and false negative rates is preferable to a single-minded concentration on false positive rates alone.

Reproducibility and Statistical Methodology

Abstract

In 2015 the Open Science Collaboration (OSC) (Nosek et al 2015) published a highly influential paper which claimed that a large fraction of published results in the psychological sciences were not reproducible. In this article we review this claim from several points of view. We first offer an extended analysis of the methods used in that study. We show that the OSC methodology induces a bias that is able by itself to explain the discrepancy between the OSC estimates of reproducibility and other more optimistic estimates made by similar studies. The article also offers a more general literature review and discussion of reproducibility in experimental science. We argue, for both scientific and ethical reasons, that a considered balance of false positive and false negative rates is preferable to a single-minded concentration on false positive rates alone.
Paper Structure (22 sections, 21 equations, 7 figures, 4 tables)

This paper contains 22 sections, 21 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. A simple mathematical model for reproducibility
  3. Model definition
  4. What value can we expect effect prevalence $\pi$ to be?
  5. Prevalence $\pi$ for clinical trials
  6. Equipoise
  7. Other interpretation of the reproducibility model
  8. On the use of preliminary data for power analyses
  9. A simple power analysis model
  10. One- versus two-sided tests
  11. Two-stage power calculation
  12. Estimation of effect size
  13. The OSC-RP revisited
  14. Other reproducibility projects
  15. Prevailing views on effect size estimation and power analyses
  16. ...and 7 more sections

Figures (7)

  • Figure 1: Decision tree representation of reproducibility model.
  • Figure 2: Contour plots of $\beta_u(\eta,t,\alpha, \beta)$ and $\beta_c(\eta,t,\alpha, \beta)$ for fixed nominal type I, II errors $\alpha = 0.05$, $\beta = 0.1$. Contour for $n^*/n = 1$ is superimposed using a dashed line [- - -]. Values $t = z_{\alpha/2}$ and $\eta = z_{\alpha/2} + z_\beta$ are indicated for convenience.
  • Figure 3: Contour plots of $R^*_u(\eta,t,\alpha, \beta)$ and $R^*_c(\eta,t,\alpha, \beta)$ for fixed nominal type I, II errors $\alpha = 0.05$, $\beta = 0.1$. Contour for $n^*/n = 1$ is indicated by a dashed line [- - -]. Values $t = z_{\alpha/2}$ and $\eta = z_{\alpha/2} + z_\beta$ are indicated for convenience.
  • Figure 4: Contour plots of $\beta_c(\eta,t,0.05, 0.05)$ and $R^*_c(\eta,t,0.05, 0.05)$. Contour for $n^*/n = 1$ is indicated by a dashed line [- - -]. Values $t = z_{\alpha/2}$ and $\eta = z_{\alpha/2} + z_\beta$ are indicated for convenience.
  • Figure 5: Maximum likelihood estimate of $\eta$ based on a single observation of $z_p \in [2.0,5.0]$, truncated at $z_p \geq t = z_{0.025}$. The identity is indicated by a dashed line [- - -].
  • ...and 2 more figures