Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Paired Seed Evaluation: Statistical Reliability for Learning-Based Simulators

Udit Sharma

TL;DR

This paper addresses high variance in evaluation of learning-based simulators due to seed-driven randomness and proposes Paired Seed Evaluation (PSE) as a principled design to exploit shared randomness. By holding the same seed across competing interventions, PSE induces within-seed differences that cancel seed-level noise, yielding variance reduction and tighter confidence intervals when seed-level correlation is positive. The authors derive explicit variance and sample-size formulas, show empirical evidence of strong positive seed-level correlations, and demonstrate substantial effective sample size gains in a TaxAI-based case study with a learned UBI policy. The work argues that PSE is a practical, broadly applicable design principle that can improve the reliability and interpretability of benchmark conclusions without additional computational cost, and it provides guidelines for when to adopt it as the default evaluation approach.

Abstract

Machine learning systems appear stochastic but are deterministically random, as seeded pseudorandom number generators produce identical realisations across executions. Learning-based simulators are widely used to compare algorithms, design choices, and interventions under such dynamics, yet evaluation outcomes often exhibit high variance due to random initialisation and learning stochasticity. We analyse the statistical structure of comparative evaluation in these settings and show that standard independent evaluation designs fail to exploit shared sources of randomness across alternatives. We formalise a paired seed evaluation design in which competing systems are evaluated under identical random seeds, inducing matched realisations of stochastic components and strict variance reduction whenever outcomes are positively correlated at the seed level. This yields tighter confidence intervals, higher statistical power, and effective sample size gains at fixed computational budgets. Empirically, seed-level correlations are typically large and positive, producing order-of-magnitude efficiency gains. Paired seed evaluation is weakly dominant in practice, improving statistical reliability when correlation is present and reducing to independent evaluation without loss of validity when it is not.

Paired Seed Evaluation: Statistical Reliability for Learning-Based Simulators

TL;DR

This paper addresses high variance in evaluation of learning-based simulators due to seed-driven randomness and proposes Paired Seed Evaluation (PSE) as a principled design to exploit shared randomness. By holding the same seed across competing interventions, PSE induces within-seed differences that cancel seed-level noise, yielding variance reduction and tighter confidence intervals when seed-level correlation is positive. The authors derive explicit variance and sample-size formulas, show empirical evidence of strong positive seed-level correlations, and demonstrate substantial effective sample size gains in a TaxAI-based case study with a learned UBI policy. The work argues that PSE is a practical, broadly applicable design principle that can improve the reliability and interpretability of benchmark conclusions without additional computational cost, and it provides guidelines for when to adopt it as the default evaluation approach.

Abstract

Machine learning systems appear stochastic but are deterministically random, as seeded pseudorandom number generators produce identical realisations across executions. Learning-based simulators are widely used to compare algorithms, design choices, and interventions under such dynamics, yet evaluation outcomes often exhibit high variance due to random initialisation and learning stochasticity. We analyse the statistical structure of comparative evaluation in these settings and show that standard independent evaluation designs fail to exploit shared sources of randomness across alternatives. We formalise a paired seed evaluation design in which competing systems are evaluated under identical random seeds, inducing matched realisations of stochastic components and strict variance reduction whenever outcomes are positively correlated at the seed level. This yields tighter confidence intervals, higher statistical power, and effective sample size gains at fixed computational budgets. Empirically, seed-level correlations are typically large and positive, producing order-of-magnitude efficiency gains. Paired seed evaluation is weakly dominant in practice, improving statistical reliability when correlation is present and reducing to independent evaluation without loss of validity when it is not.
Paper Structure (31 sections, 2 theorems, 29 equations, 3 figures, 3 tables)

This paper contains 31 sections, 2 theorems, 29 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Evaluation Practice in Learning-Based Simulators
  3. Paired Seed Evaluation
  4. Experimental Design and Estimand
  5. Independent and Paired Evaluation Designs
  6. Variance Reduction under Paired Evaluation
  7. Interpretation and Corollary
  8. Empirical Evidence of Positive Seed-Level Correlation
  9. Implications for Evaluation Sample Size
  10. Scaling of Standard Error with Sample Size
  11. Confidence Interval Shrinkage
  12. Statistical Power to Detect Policy Effects
  13. Directional Stability
  14. Effective Sample Size
  15. Implications and Summary
  16. ...and 16 more sections

Key Result

Theorem 1

Let $Y(1,s)$ and $Y(0,s)$ be square-integrable random variables. The variance of the paired estimator satisfies

Figures (3)

  • Figure 1: 95% Confidence Interval half-width of the estimated policy effect under paired and independent evaluation designs for the GDP (Solid) and Gini (Dashed) objective, using wealth Gini as the outcome metric.
  • Figure 2: Statistical Power of the estimated policy effect under paired and independent evaluation designs for the GDP (Solid) and Gini (Dashed) objective, to detect a 2% change in wealth Gini.
  • Figure 3: Sign Agreement of the estimated policy effect under paired and independent evaluation designs, for the GDP (Solid) and Gini (Dashed) objectives, using wealth gini as the outcome metric.

Theorems & Definitions (3)

  • Theorem 1
  • proof
  • Corollary 1