Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Testing the statistical compatibility of independent data sets

M. Maltoni, T. Schwetz

TL;DR

The paper tackles the limitation of standard goodness-of-fit in global, multi-parameter analyses by introducing the parameter goodness-of-fit (PG) test, which assesses compatibility across $D$ independent data sets within a single model. It derives the probability distribution of the PG statistic $\bar{\chi}^2_{\min}$ and shows it follows a $\chi^2$ distribution with $P_c$ degrees of freedom, analogous to the SG case, enabling exact p-values. Through theoretical proofs and concrete examples—including a two-experiment one-parameter scenario and a neutrino-oscillation analysis—the authors demonstrate that PG avoids dilution by data points insensitive to the crucial parameters and can reveal inter-set tensions that SG maymiss. The pull approach is discussed to extend PG to correlated data via nuisance parameters with penalty terms, broadening its applicability to realistic analyses. Overall, PG provides a principled, complementary framework for testing cross-experiment consistency in large, multi-parameter fits with practical impact for neutrino physics and beyond, where independent data subsets are common.

Abstract

We discuss a goodness-of-fit method which tests the compatibility between statistically independent data sets. The method gives sensible results even in cases where the chi^2-minima of the individual data sets are very low or when several parameters are fitted to a large number of data points. In particular, it avoids the problem that a possible disagreement between data sets becomes diluted by data points which are insensitive to the crucial parameters. A formal derivation of the probability distribution function for the proposed test statistic is given, based on standard theorems of statistics. The application of the method is illustrated on data from neutrino oscillation experiments, and its complementarity to the standard goodness-of-fit is discussed.

Testing the statistical compatibility of independent data sets

TL;DR

The paper tackles the limitation of standard goodness-of-fit in global, multi-parameter analyses by introducing the parameter goodness-of-fit (PG) test, which assesses compatibility across independent data sets within a single model. It derives the probability distribution of the PG statistic and shows it follows a distribution with degrees of freedom, analogous to the SG case, enabling exact p-values. Through theoretical proofs and concrete examples—including a two-experiment one-parameter scenario and a neutrino-oscillation analysis—the authors demonstrate that PG avoids dilution by data points insensitive to the crucial parameters and can reveal inter-set tensions that SG maymiss. The pull approach is discussed to extend PG to correlated data via nuisance parameters with penalty terms, broadening its applicability to realistic analyses. Overall, PG provides a principled, complementary framework for testing cross-experiment consistency in large, multi-parameter fits with practical impact for neutrino physics and beyond, where independent data subsets are common.

Abstract

We discuss a goodness-of-fit method which tests the compatibility between statistically independent data sets. The method gives sensible results even in cases where the chi^2-minima of the individual data sets are very low or when several parameters are fitted to a large number of data points. In particular, it avoids the problem that a possible disagreement between data sets becomes diluted by data points which are insensitive to the crucial parameters. A formal derivation of the probability distribution function for the proposed test statistic is given, based on standard theorems of statistics. The application of the method is illustrated on data from neutrino oscillation experiments, and its complementarity to the standard goodness-of-fit is discussed.

Paper Structure

This paper contains 14 sections, 2 theorems, 49 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Goodness-of-fit tests
  3. The standard goodness-of-fit
  4. The parameter goodness-of-fit
  5. The probability distribution function of $\bar{\chi}^2_\mathrm{min}$
  6. The standard goodness-of-fit
  7. The parameter goodness-of-fit
  8. Examples and discussion
  9. The determination of one parameter by two experiments
  10. Consistency of PG and SG for independent data points
  11. Application to neutrino oscillation data
  12. General remarks on the PG
  13. Correlations due to theoretical uncertainties
  14. Conclusions

Key Result

Proposition 1

Let $\hat{\hbox{\boldmath$\theta$}}$ be the values of the parameters which minimize Eq. eq:chi2. Then with $\chi^2_\mathrm{min} = \hat{ {\rm\bf y} }^T \hat{ {\rm\bf y} }$ and $\hat{ {\rm\bf y} } \equiv {\rm\bf y} (\hat{\hbox{\boldmath$\theta$}})$, has a $\chi^2$-distribution with $N-P$d.o.f. and $\Delta\chi^2$ has a $\chi^2$-distribution with $P$d.o.f. and is statistically independent of $\chi^2_

Figures (2)

  • Figure 1: Parameter structure of the three data sets from reactor, solar and atmospheric neutrino experiments.
  • Figure 2: PG for solar and atmospheric neutrino data in the (2+2) scheme.

Theorems & Definitions (2)

  • Proposition 1
  • Proposition 2