A Unified Approach to the Classical Statistical Analysis of Small Signals

TL;DR

The paper tackles the problem that traditional classical upper-limit procedures can yield unphysical or miscovered intervals when the data influence the reporting choice. It introduces a unified Neyman confidence belt constructed with a likelihood-ratio ordering to ensure nonempty intervals with correct coverage, eliminating flip-flop shortcomings. The method is demonstrated on Poisson processes with known background, Gaussian measurements with a physical-boundary, and extended to multidimensional neutrino-oscillation searches, where it provides improved power while maintaining coverage compared to standard classical methods and Bayesian approaches. It also advocates reporting an experiment's sensitivity alongside limits to contextualize results, promoting transparent and reproducible statistical reporting in high-energy physics.

Abstract

We give a classical confidence belt construction which unifies the treatment of upper confidence limits for null results and two-sided confidence intervals for non-null results. The unified treatment solves a problem (apparently not previously recognized) that the choice of upper limit or two-sided intervals leads to intervals which are not confidence intervals if the choice is based on the data. We apply the construction to two related problems which have recently been a battle-ground between classical and Bayesian statistics: Poisson processes with background, and Gaussian errors with a bounded physical region. In contrast with the usual classical construction for upper limits, our construction avoids unphysical confidence intervals. In contrast with some popular Bayesian intervals, our intervals eliminate conservatism (frequentist coverage greater than the stated confidence) in the Gaussian case and reduce it to a level dictated by discreteness in the Poisson case. We generalize the method in order to apply it to analysis of experiments searching for neutrino oscillations. We show that this technique both gives correct coverage and is powerful, while other classical techniques that have been used by neutrino oscillation search experiments fail one or both of these criteria.

Figures (15)

• Figure 1: A generic confidence belt construction and its use. For each value of $\mu$, one draws a horizontal acceptance interval $[x_1,x_2]$ such that $P(x\in [x_1,x_2]\,|\mu) = \alpha$. Upon performing an experiment to measure $x$ and obtaining the value $x_0$, one draws the dashed vertical line through $x_0$. The confidence interval $[\mu_1,\mu_2]$ is the union of all values of $\mu$ for which the corresponding acceptance interval is intercepted by the vertical line.
• Figure 2: Standard confidence belt for 90% C.L. upper limits for the mean of a Gaussian, in units of the rms deviation. The second line in the belt is at $x=+\infty$.
• Figure 3: Standard confidence belt for 90% C.L. central confidence intervals for the mean of a Gaussian, in units of the rms deviation.
• Figure 4: Plot of confidence belts implicitly used for 90% C.L. confidence intervals (vertical intervals between the belts) quoted by flip-flopping Physicist X, described in the text. They are not valid confidence belts, since they can cover the true value at a frequency less than the stated confidence level. For $1.36<\mu<4.28$, the coverage (probability contained in the horizontal acceptance interval) is 85%.
• Figure 5: Standard confidence belt for 90% C.L. upper limits, for unknown Poisson signal mean $\mu$ in the presence of Poisson background with known mean $b=3.0$. The second line in the belt is at $n=+\infty$.