Data Reliability Scoring

TL;DR

This work addresses the challenge of assessing data reliability when ground truth is unobserved by introducing ground-truth-based reliability orderings and the Gram determinant reliability score $\Gamma$, which remains invariant to the unknown data-generation experiment. By relating true data, reported data, and auxiliary observations through a misreport matrix $\mathbf{Q}$ and an unknown experiment $\mathbf{P}$, the authors formalize a detail-free scoring framework and prove impossibility results that motivate the need for a robust, experiment-agnostic score. The Gram determinant score exploits the geometric volume spanned by joint observation-distribution vectors, yielding an invariant ranking up to scaling: $\Gamma(\mathbf{P}\mathbf{Q}) = \det(\mathbf{P}^{\top}\mathbf{P})\det(\mathbf{Q})^2$. They provide plug-in and stratified-matching estimators with finite-sample guarantees and extend the construction to kernels to handle continuous or structured observation spaces; empirical results on synthetic data, CIFAR-10 embeddings, and employment data demonstrate that the score reliably reflects data quality across diverse observation processes and misreporting patterns.

Abstract

How can we assess the reliability of a dataset without access to ground truth? We introduce the problem of reliability scoring for datasets collected from potentially strategic sources. The true data are unobserved, but we see outcomes of an unknown statistical experiment that depends on them. To benchmark reliability, we define ground-truth-based orderings that capture how much reported data deviate from the truth. We then propose the Gram determinant score, which measures the volume spanned by vectors describing the empirical distribution of the observed data and experiment outcomes. We show that this score preserves several ground-truth based reliability orderings and, uniquely up to scaling, yields the same reliability ranking of datasets regardless of the experiment -- a property we term experiment agnosticism. Experiments on synthetic noise models, CIFAR-10 embeddings, and real employment data demonstrate that the Gram determinant score effectively captures data quality across diverse observation processes.

Figures (5)

• Figure 1: Gram determinant scores and parallelepipeds. The Gram determinant score of true data, $\Gamma({\bm{P}}{\bm{Q}}_{\bm{x}})$, is the squared volume of the blue parallelepiped spanned by column vectors in ${\bm{P}}{\bm{Q}}_{\bm{x}}$, $vol({\bm{P}}{\bm{Q}}_{\bm{x}})^2$. As ${\Gamma({\bm{P}}{\bm{Q}})}= {\Gamma({\bm{P}}{\bm{Q}}_{\bm{x}}{\bm{Q}}_{\hat{{\bm{x}}}|{\bm{x}}}^\intercal)} = {\Gamma({\bm{P}}{\bm{Q}}_{\bm{x}}{\bm{Q}}_{\hat{{\bm{x}}}|{\bm{x}}})}$, the Gram determinant score of reported data is the squared volume of the red parallelepiped, $vol({\bm{P}}{\bm{Q}}_{\bm{x}} {\bm{Q}}_{\hat{{\bm{x}}}|{\bm{x}}})^2$, which is smaller than that of the true data because ${\bm{Q}}_{\hat{{\bm{x}}}|{\bm{x}}}$ is column stochastic and each column of ${\bm{P}}{\bm{Q}}_{\bm{x}} {\bm{Q}}_{\hat{{\bm{x}}}|{\bm{x}}}$ is a convex combination of columns of ${\bm{P}}{\bm{Q}}_{\bm{x}}$.
• Figure 2: Gram determinant reliability score on categorical synthetic data.
• Figure 3: Matched rankings on categorical synthetic data.
• Figure 4: Gram determinant reliability for image–label experiments under six manipulation policies
• Figure 5: Comparison of the Top-$k$ volume, Max correlation, KL divergence, and $\chi^2$-mutual information scores under different corruption levels and metrics.

Theorems & Definitions (50)

• Proposition 2.1: Refinement
• Definition 2.2
• Proposition 3.1
• Definition 4.1
• Example 4.2
• Theorem 4.3
• Proposition 4.4
• Definition 4.5: plug-in Gram determinant reliability score
• Proposition 4.6
• Definition 4.7