Functional Linear Regression of Cumulative Distribution Functions

TL;DR

This work develops a functional linear regression framework for contextual CDFs by positing $F(x,t)=\theta_*^\top\Phi(x,t)$ with $\theta_*$ on the simplex. It introduces ridge-based estimators and establishes nonasymptotic, self-normalized error bounds that scale as $\tilde{O}(\sqrt{d/n})$ across fixed, random, and adversarial designs, and proves matching minimax lower bounds. The authors also provide burn-in-free estimation variants, analyze mismatched data-generation scenarios, and generalize to infinite-dimensional Hilbert spaces, yielding bounds that reduce to the finite-dimensional rates when appropriate. Comprehensive numerical experiments on synthetic and real data corroborate the theoretical rates and demonstrate practical utility for risk assessment and distributional analysis. The framework thus offers a principled, statistically optimal approach to learning and evaluating context-conditioned CDFs in diverse settings.

Abstract

The estimation of cumulative distribution functions (CDF) is an important learning task with a great variety of downstream applications, such as risk assessments in predictions and decision making. In this paper, we study functional regression of contextual CDFs where each data point is sampled from a linear combination of context dependent CDF basis functions. We propose functional ridge-regression-based estimation methods that estimate CDFs accurately everywhere. In particular, given $n$ samples with $d$ basis functions, we show estimation error upper bounds of $\widetilde O(\sqrt{d/n})$ for fixed design, random design, and adversarial context cases. We also derive matching information theoretic lower bounds, establishing minimax optimality for CDF functional regression. Furthermore, we remove the burn-in time in the random design setting using an alternative penalized estimator. Then, we consider agnostic settings where there is a mismatch in the data generation process. We characterize the error of the proposed estimators in terms of the mismatched error, and show that the estimators are well-behaved under model mismatch. Moreover, to complete our study, we formalize infinite dimensional models where the parameter space is an infinite dimensional Hilbert space, and establish a self-normalized estimation error upper bound for this setting. Notably, the upper bound reduces to the $\widetilde O(\sqrt{d/n})$ bound when the parameter space is constrained to be $d$-dimensional. Our comprehensive numerical experiments validate the efficacy of our estimation methods in both synthetic and practical settings.

Figures (7)

• Figure 1: A visualization of the data generating process. For each $j\in[6]$ with context $x^{(j)}\in\mathcal{X}$, the upper row shows the $d$ contextual CDFs ($\phi_i(x^{(j)},\cdot),\ i\in[d]$) under the context $x^{(j)}$. For $y^{(j)}$ drawn from the CDF$F(x^{(j)},\cdot)=\theta_*^\top\Phi(x^{(j)},\cdot)$ where $\Phi(x^{(j)},\cdot):=[\phi_i(x^{(j)},\cdot),\dots,\phi_d(x^{(j)},\cdot)]^\top$, the bottom row shows the sample empirical CDF$\textup{I}_{y^{(j)}}(\cdot):=\mathbbm{1}\{y^{(j)}\le \cdot\}$.
• Figure 2: Means and 90% confidence intervals of un-normalized $\ell^2$-errors $\|\widehat{\theta}_{\lambda}-\theta_*\|$, KS distances $\textup{KS}(\widehat{F}_{\lambda}(x,\cdot),F(x,\cdot))$, and self-normalized errors $\|\widehat{\theta}_{\lambda}-\theta_*\|_{U_n(\lambda)}$ against sample size $n$ in logarithmic scale in Bernoulli synthetic data experiments.
• Figure 3: Means and 90% confidence intervals of un-normalized $\ell^2$-errors $\|\widehat{\theta}_{\lambda}-\theta_*\|$, KS distances $\textup{KS}(\widehat{F}_{\lambda}(x,\cdot),F(x,\cdot))$, and self-normalized errors $\|\widehat{\theta}_{\lambda}-\theta_*\|_{U_n(\lambda)}$ against $d/\mu_{\min}(U_n(\lambda))$ and dimension $d$ in logarithmic scale in Bernoulli synthetic data experiments.
• Figure 4: Means and 90% confidence intervals of un-normalized $\ell^2$-errors $\|\widehat{\theta}_{\lambda}-\theta_*\|$, KS distances $\textup{KS}(\widehat{F}_{\lambda}(x,\cdot),F(x,\cdot))$, and self-normalized errors $\|\widehat{\theta}_{\lambda}-\theta_*\|_{U_n(\lambda)}$ and $\|\widehat{\theta}_{\lambda}-\theta_*\|_{\Sigma_n}$ against sample size $n$ in logarithmic scale in polynomial CDF synthetic data experiments.
• Figure 5: Means and 90% confidence intervals of un-normalized $\ell^2$-errors $\|\widehat{\theta}_{\lambda}-\theta_*\|$, KS distances $\textup{KS}(\widehat{F}_{\lambda}(x,\cdot),F(x,\cdot))$, and self-normalized errors $\|\widehat{\theta}_{\lambda}-\theta_*\|_{U_n(\lambda)}$ and $\|\widehat{\theta}_{\lambda}-\theta_*\|_{\Sigma_n}$ against dimension $d$ in logarithmic scale in polynomial CDF synthetic data experiments.

Theorems & Definitions (45)

• Theorem 1: Self-normalized bound in adversarial setting
• Proposition 2: Self-normalized bound in adversarial setting for unregularized estimator
• Corollary 3: Self-normalized bound in fixed design setting
• Theorem 4: Self-normalized bound in random design setting
• Proposition 5: Self-normalized bound in random design setting for regularized estimator
• Lemma 6
• Theorem 7: Self-normalized bound in random setting without burn-in time
• Theorem 8: Information theoretic lower bound in adversarial setting
• Corollary 9: Information theoretic lower bound in random setting
• Theorem 10: Self-normalized bound in mismatched adversarial setting