Many Wrongs Make a Right: Leveraging Biased Simulations Towards Unbiased Parameter Inference

Abstract

In particle physics, as in many areas of science, parameter inference relies on simulations to bridge the gap between theory and experiment. Recent developments in simulation-based inference have boosted the sensitivity of analyses; however, biases induced by simulation-data mismodeling can be difficult to control within standard inference pipelines. In this work, we propose a Template-Adapted Mixture Model to confront this problem in the context of signal fraction estimation: inferring the population proportion of signal in a mixed sample of signal and background, both of which follow arbitrarily complex distributions. We harness many biased simulations to perform data-driven estimates of each process distribution in the signal region, substantially reducing the bias on the signal fraction due to the domain shift between simulation and reality. We explore different methodological choices, including model selection, feature representation, and statistical method, and apply them to a Gaussian toy example and to a semi-realistic di-Higgs measurement. We find that the presented methods successfully leverage the biased simulations to provide estimates with well-calibrated uncertainties.

Figures (18)

• Figure 1: A schematic representation of the domain shift problem which we seek to address. The left panel corresponds to the signal and the right to the background. Each panel shows three MSDs and the corresponding TD. The black lines connecting the MSDs represent the possibility that the MSDs may (or may not) be generated through variations of continuous nuisance parameters, but that we have in mind the scenario where there is no value of these nuisance parameters exactly corresponding to the real TDs. The blue arrows show how the MSDs are utilized to form the best-fit signal and background models, $\hat{s}(x)$ and $\hat{b}(x)$, which are likely to be closer to the true signal and background distributions than any of the individual MSDs. The bottom line emphasizes that the best-fit signal fraction $\hat{\kappa}$, signal model $\hat{s}(x)$, and background model $\hat{b}(x)$ are simultaneously inferred from the TD. Black quantities are the observed quantities that we have direct access to through simulation or observation, red quantities are the truth that we hope to discover through the data, and blue quantities are the results of our modeling.
• Figure 2: Gaussian case study TD dataset, showing $20\%$ of the sample. $50$k background (red) points are distributed according to $\mathcal{N}( \mu_b,C_b)$ and $5$k signal (blue) points are distributed according to $\mathcal{N}( \mu_s, C_s)$. The MSDs for signal and background are biased and never match the true distributions, as discussed in the text. The black lines show the necessary binning for Bayesian Topic Modeling. Both Frequentist Neural Estimation and Bayesian Topic Modeling restrict the TD and MSDs to the fiducial region corresponding to the outside edges of the outermost bins.
• Figure 3: Coverage performance plot for the Gaussian case using Frequentist Neural Estimation with Wald intervals in the left panel and profile intervals in the right panel, for a varying number of MSDs. The solid gray line in each panel shows the performance of the baseline model, while the dashed black line shows the nominal behavior where the observed and expected coverage are equal. Each colorful line corresponds to a model with a different number $K$ of MSDs each for the signal and background. Each solid line corresponds to an average over $30$ choices of MSDs, and the spread depicted corresponds to the standard error in that average.
• Figure 4: A $2$D histogram of the estimated uncertainty in $\kappa$, $\sigma_\kappa \equiv \sqrt{C^{\kappa \kappa}}$ versus the pull in $\kappa$, $z_\text{Wald}$, for Frequentist Neural Estimation with $K=10$ MSDs for the Gaussian case. The top (right) panel shows the marginal distribution of $z_\text{Wald}$ ($\sigma_\kappa$). Blue is the $K=10$ model, gray is the baseline, and the dashed black line in the top panel corresponds to the nominal standard normal distribution of $z_\text{Wald}$.
• Figure 5: In black, the distribution of Hellinger distances obtained from comparing the MSDs to their corresponding TD signal (left) and background (right) for the Gaussian case. In blue, the distribution of Hellinger distances obtained from comparing the learned $\hat{s}$ and $\hat{b}$ from all pseudo-experiments with their corresponding TD signal (left) and background (right) distributions.