A Bayesian Approach to Estimating Effect Sizes in Educational Research

Abstract

In this paper, we demonstrate a purely Bayesian approach for estimating within-group and between-group effect sizes for learning outcomes encountered in educational research, taking naturally into account the multilevel structure of the data, as well as heterogeneous residual variances among time points and conditions. We provide a detailed implementation using the brms package in R serving as a wrapper for the probabilistic programming language Stan. We recommend that for a pooled design, one computes an effect size $d_s$ similar to a Cohen's $d$, and for a paired design, one should compute two possibly different quantities $d_s$ and $d_z$ to correct for correlations in within-group designs and allow for comparability across different studies. All these effect sizes are based on ideas coming from Hedge's total effect size $δ_t$ introduced in 2007. Ultimately, these estimates allow us to study the differential effectiveness of educational interventions with respect to classes.

Figures (6)

• Figure 1: Distributions of Solution Rates in a Pretest-Posttest Design With Respect to Condition
• Figure 2: Graphical Posterior Predictive Check for the Posterior Distribution of the Score Generated by the Bayesian ANOVA Model Without Heterogeneous Residual Variances
• Figure 3: Graphical Posterior Predictive Check for the Posterior Distribution of the Score Generated by the Bayesian ANOVA Model With Heterogeneous Residual Variances
• Figure 4: Plot of the Mean Residual Error of the Score Generated by the Bayesian ANOVA Model With Heterogeneous Residual Variances
• Figure 5: Histogram of Posterior Samples of the Effect Size $d_s$