Likelihood-Preserving Embeddings for Statistical Inference

TL;DR

This work defines likelihood-preserving embeddings that substitute raw data in likelihood-based inference by controlling the likelihood-ratio distortion $\Delta_n$. The Hinge Theorem establishes that if the pointwise embedding error $\varepsilon_n$ satisfies $\varepsilon_n=o_p(1/n)$, then all standard inference—LR tests, Bayes factors, MLEs, and information criteria like AIC/BIC—are preserved, while a universal preservation result is impossible without essentially invertible embeddings. The authors provide a constructive neural framework to learn approximate sufficient statistics, with explicit bounds connecting training loss to inferential guarantees, and validate the theory through Gaussian, Cauchy, synthetic Gaussian mixtures, and a distributed multi-site clinical trial application. The results offer a principled pathway to perform exact likelihood-based inference under privacy, bandwidth, or computational constraints, by learning representations that retain the full inferential content of the data. Overall, the paper bridges classical sufficiency theory and modern representation learning, delivering practical tools for inference with compressed, privacy-preserving data.

Abstract

Modern machine learning embeddings provide powerful compression of high-dimensional data, yet they typically destroy the geometric structure required for classical likelihood-based statistical inference. This paper develops a rigorous theory of likelihood-preserving embeddings: learned representations that can replace raw data in likelihood-based workflows -- hypothesis testing, confidence interval construction, model selection -- without altering inferential conclusions. We introduce the Likelihood-Ratio Distortion metric $Δ_n$, which measures the maximum error in log-likelihood ratios induced by an embedding. Our main theoretical contribution is the Hinge Theorem, which establishes that controlling $Δ_n$ is necessary and sufficient for preserving inference. Specifically, if the distortion satisfies $Δ_n = o_p(1)$, then (i) all likelihood-ratio based tests and Bayes factors are asymptotically preserved, and (ii) surrogate maximum likelihood estimators are asymptotically equivalent to full-data MLEs. We prove an impossibility result showing that universal likelihood preservation requires essentially invertible embeddings, motivating the need for model-class-specific guarantees. We then provide a constructive framework using neural networks as approximate sufficient statistics, deriving explicit bounds connecting training loss to inferential guarantees. Experiments on Gaussian and Cauchy distributions validate the sharp phase transition predicted by exponential family theory, and applications to distributed clinical inference demonstrate practical utility.

Figures (4)

• Figure 1: Pointwise Framework Validation. Empirical demonstration of the relationship $\Delta_n \leq 2n\varepsilon_n$ for Gaussian $\mathcal{N}(\mu,\sigma^2)$ with $n=100$. Left: Pointwise approximation error $\varepsilon_n$ (per-sample log-likelihood error) drops from $O(1)$ at $m=1$ (incomplete) to machine precision at $m=2$ (sufficient dimension). Right: Comparison of ratio distortion $\Delta_n$ (actual, blue bars) with theoretical bound $2n\varepsilon_n$ (orange bars). The bound is tight for incomplete embeddings (ratio $\approx 0.43$) and becomes vacuous at exact sufficiency, validating that minimizing pointwise error $\varepsilon_n$ automatically controls ratio distortion $\Delta_n$.
• Figure 2: Gaussian and Cauchy: Exact Sufficiency vs. Approximation Trade-off. Each panel shows both pointwise approximation error $\varepsilon_n$ (left, circles) and likelihood-ratio distortion $\Delta_n$ (right, squares) per sample on log scale. Gaussian$\mathcal{N}(\mu, \sigma^2)$ (top): Sharp phase transition at $m=2$ (the sufficient dimension). Both $\varepsilon_n$ and $\Delta_n$ drop from $O(1)$ to machine precision ($\approx 10^{-15}$)---a 14-order-of-magnitude decrease---validating exact sufficiency. Beyond $m=2$, additional dimensions provide no benefit. Cauchy$\text{Cauchy}(\theta, 1)$ (bottom): Smooth monotonic decay for both metrics without reaching zero, validating the Pitman-Koopman-Darmois theorem that non-exponential families lack finite-dimensional sufficient statistics. Evaluation uses 100 independent datasets of $n=100$ samples each.
• Figure 3: Synthetic GMM Validation. Panel A: Scatter plot of true vs. surrogate log-likelihoods for 50 random parameter perturbations of a 3-component Gaussian mixture model, showing near-perfect agreement after linear calibration ($r = 0.987$). The neural embedding achieves $\varepsilon_n = 0.11$ (pointwise approximation error per sample) and $\Delta_n = 0.21$ (likelihood-ratio distortion per sample), satisfying the theoretical bound $\Delta_n \leq 2n\varepsilon_n$. Panel B: Scatter plot of true vs. surrogate likelihood ratios for all $\binom{50}{2} = 1225$ pairs, demonstrating tight preservation ($r = 0.987$). The neural embedding successfully compresses 1000 samples in $\mathbb{R}^{10}$ into a 16-dimensional summary while preserving likelihood-based inference.
• Figure 4: Multi-Site Clinical Trial. Statistical power for testing treatment effect ($H_0: \beta = 0$) across methods in a 5-site trial with 200 patients per site. Summary-based (16 numbers/site) achieves identical power to the pooled gold standard; compressed (8 numbers/site) achieves 99% relative efficiency, demonstrating near-perfect information preservation. Meta-analysis loses 50% relative power. Each site transmits only 8 numbers instead of 800 patient measurements---a 100-fold data reduction with $<$1% power loss. Shaded regions show 95% confidence intervals based on 500 independent simulated trials.

Theorems & Definitions (35)

• Definition 2.5: Embedding
• Definition 2.6: Dataset Embedding
• Remark 2.7
• Definition 2.8: Decoder
• Definition 2.9: Surrogate Likelihood
• Definition 2.10: Pointwise Approximation Error
• Definition 2.11: $\varepsilon$-Sufficient Embedding
• Remark 2.12: Why $o_p(1/n)$?
• Proposition 2.13: Pointwise Implies Ratio Preservation
• proof