Optimized Model Selection for Estimating Treatment Effects from Costly Simulations of the US Opioid Epidemic

TL;DR

Different methods to explore what model works best at a specific sample size are discussed and a mathematical analysis of the Mean Squared Error (MSE) equation and how its components decide which model to select and why a specific method behaves that way in a range of sample sizes is provided.

Abstract

Agent-based simulation with a synthetic population can help us compare different treatment conditions while keeping everything else constant within the same population (i.e., as digital twins). Such population-scale simulations require large computational power (i.e., CPU resources) to get accurate estimates for treatment effects. We can use meta models of the simulation results to circumvent the need to simulate every treatment condition. Selecting the best estimating model at a given sample size (number of simulation runs) is a crucial problem. Depending on the sample size, the ability of the method to estimate accurately can change significantly. In this paper, we discuss different methods to explore what model works best at a specific sample size. In addition to the empirical results, we provide a mathematical analysis of the MSE equation and how its components decide which model to select and why a specific method behaves that way in a range of sample sizes. The analysis showed why the direction estimation method is better than model-based methods in larger sample sizes and how the between-group variation and the within-group variation affect the MSE equation.

Figures (5)

• Figure 1: Model selection in costly sample regimes. This figure shows which model will have the lowest MSE, given the sample size. The arrow points in the direction of increasing sample size, and at each interval, the equation specified is for the model that achieves the least MSE. Notice the increasing complexity of the optimal model with increasing sample size. With a large enough number of samples directly estimating each treatment condition is optimum.
• Figure 2: State transition diagram for the OUD model.
• Figure 3: The MSE values for different models compared to sample sizes of 100, 200, 300, and 400, respectively, where it show that simpler model-based methods are better than higher terms model-based methods and model-free methods.
• Figure 4: The MSE values for different models compared to sample sizes from 500 to 6000 where show that model-based methods perform well until a specific sample size that the model-free method supersedes in performance.
• Figure 5: The main components that decide the MSE equation and affect the choice of a model. This figure shows the main variables that comprise the MSE value. The first one, $\rho^2$, is the between-group variation where its increase will affect the bias part of the MSE equation for the model-based method. The second variable (in clockwise order) $L$ is the number of levels, though, in this problem, we defined a fixed number of levels, but the idea generalized as $L\to\infty$ and \ref{['eq:nstar']} for $n^{\star}$ is increasing in $L$ for large $L$. In those regimes, we prefer to use model-based methods over a broader range because the within-group variability term dominates the MSE equation as $L\to\infty$. The third factor affecting our model selection is $\sigma^2$ or within-group variability; with increasing $\sigma^2$, we prefer to use more and more complex models to minimize MSE. Lastly, $n$ is the sample size which is the most critical factor in optimizing model selection (\ref{['fig:n_axis']}). As $n \to \infty$, the variance terms go to zero, and the bias part in the model-based MSE equation becomes dominant, at which point direct estimation is preferred (there are no advantages in the use of models in large sample regimes).

Theorems & Definitions (1)

• Theorem 1