Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multilevel Metamodels: Enhancing Inference, Interpretability, and Generalizability in Monte Carlo Simulation Studies

Joshua Gilbert, Luke Miratrix

TL;DR

The paper addresses how to robustly summarize Monte Carlo simulation results when multiple models are applied to the same simulated data. It advocates multilevel metamodels (MLMMs) to account for dependence across models and to enable generalizability assessments via random effects and empirical Bayes ranges. Through a running covariate-adjustment example in RCTs, the authors demonstrate that two-level MLMMs improve precision over OLS for aggregated results, while three-level MLMMs extending to individual results provide richer insights into consistency across simulation conditions and generalizability. The work highlights practical benefits for interpreting simulation findings, guiding model choice, and informing statistical practice, while noting limitations and avenues for future research, including Bayesian approaches and preregistration.

Abstract

Metamodels, or the regression analysis of Monte Carlo simulation results, provide a powerful tool to summarize simulation findings. However, an underutilized approach is the multilevel metamodel (MLMM) that accounts for the dependent data structure that arises from fitting multiple models to the same simulated data set. In this study, we articulate the theoretical rationale for the MLMM and illustrate how it can improve the interpretability of simulation results, better account for complex simulation designs, and provide new insights into the generalizability of simulation findings.

Multilevel Metamodels: Enhancing Inference, Interpretability, and Generalizability in Monte Carlo Simulation Studies

TL;DR

The paper addresses how to robustly summarize Monte Carlo simulation results when multiple models are applied to the same simulated data. It advocates multilevel metamodels (MLMMs) to account for dependence across models and to enable generalizability assessments via random effects and empirical Bayes ranges. Through a running covariate-adjustment example in RCTs, the authors demonstrate that two-level MLMMs improve precision over OLS for aggregated results, while three-level MLMMs extending to individual results provide richer insights into consistency across simulation conditions and generalizability. The work highlights practical benefits for interpreting simulation findings, guiding model choice, and informing statistical practice, while noting limitations and avenues for future research, including Bayesian approaches and preregistration.

Abstract

Metamodels, or the regression analysis of Monte Carlo simulation results, provide a powerful tool to summarize simulation findings. However, an underutilized approach is the multilevel metamodel (MLMM) that accounts for the dependent data structure that arises from fitting multiple models to the same simulated data set. In this study, we articulate the theoretical rationale for the MLMM and illustrate how it can improve the interpretability of simulation results, better account for complex simulation designs, and provide new insights into the generalizability of simulation findings.
Paper Structure (16 sections, 6 equations, 7 figures, 5 tables)

This paper contains 16 sections, 6 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. The Simulation Metamodel
  3. Illustrating Metamodeling Strategies Through a Simple Simulation
  4. Metamodeling Approaches
  5. Two-Level MLMMs for the Aggregated Results
  6. Three-Level MLMM for the Individual Results
  7. Contrasting the two-level and three-level MLMM
  8. Methods
  9. Results
  10. Power
  11. Validity (Type I Error)
  12. Squared Error
  13. Bias, Coverage, True Standard Error, Estimated Standard Error, and Standard Error Calibration
  14. Summary and Review
  15. Discussion
  16. ...and 1 more sections

Figures (7)

  • Figure 1: Distributions of Statistical Power by Simulation Condition
  • Figure 2: Distribution of Empirical Bayes Estimates for the Effect of "Adjusted" on Statistical Power in Model 1
  • Figure 3: Conjoint Plot of Metamodel Results for Statistical Power
  • Figure 4: Distributions of False Positive Rates by Simulation Condition
  • Figure 5: Conjoint Plot of Metamodel Results for False Positive Rates
  • ...and 2 more figures