StatTestCalculator: A New General Tool for Statistical Analysis in High Energy Physics

TL;DR

StatTestCalculator addresses the need for a general, lightweight statistical tool in high energy physics that can deliver both fast analytic estimates and exact Monte Carlo results. It implements a likelihood-based framework with nuisance parameters, using profile likelihood ratios $q_\mu$ for discovery and exclusion, and provides analytic formulas for $Z_{disc}$ and $Z_{excl}$ that account for background systematics via a relative uncertainty parameter $\delta$. The approach is extended to multi-bin analyses and validated against the CMS Combine tool, demonstrating consistent results and practical performance. STC is implemented in Python with a modular design, enabling custom test statistics, shapes, and systematics, and is released as an open-source resource for fast sensitivity studies, cross-checks, and education in HEP statistics. $Z_{disc}$, $Z_{excl}$, $q_0$, and $q_\mu$ play central roles in the framework, providing clear metrics for discovery potential and exclusion power while accommodating systematic uncertainties through analytical and MC-based methods.

Abstract

We present StatTestCalculator (STC), a new open-source statistical analysis tool designed for analysis high energy physics experiments. STC provides both asymptotic calculations and Monte Carlo simulations for computing the exact statistical significance of a discovery or for setting upper limits on signal model parameters. We review the underlying statistical formalism, including profile likelihood ratio test statistics for discovery and exclusion hypotheses, and the asymptotic distributions that allow quick significance estimates. We explain the relevant formulas for the likelihood functions, test statistic distributions, and significance metrics (both with and without incorporating systematic uncertainties). The implementation and capabilities of STC are described, and we validate its performance against the widely-used CMS Combine tool. We find excellent agreement in both the expected discovery significances and upper limit calculations. STC is a flexible framework that can accommodate systematic uncertainties and user-defined statistical models, making it suitable for a broad range of analyses.

Figures (4)

• Figure 1: The example of Monte Carlo simulated distribution of test statistic $q_0$
• Figure 2: The convergence of $\mu$ parameter for each iteration of Monte Carlo simulations through the specified grid. The set confidence level of 95% is reached for the value of $\mu \approx 1.7$.
• Figure 3: Comparison of STC calculations of the significance of discovery made with asymptotic formulae (a, c) and using Monte Carlo simulations (b,d) with and without the presence of relative systematic uncertainty
• Figure 4: Comparison of STC calculations of expected upper limits made with asymptotic formulae (a, c) and using Monte Carlo simulations (b,d) with and without the presence of relative systematic uncertainty