End-to-End Probabilistic Framework for Learning with Hard Constraints

TL;DR

ProbHardE2E optimizes a strictly proper scoring rule, without making any distributional assumptions on the target, which enables it to obtain robust distributional estimates, and can incorporate a range of non-linear constraints (increasing the power of modeling and flexibility).

Abstract

We present ProbHardE2E, a probabilistic forecasting framework that incorporates hard operational/physical constraints, and provides uncertainty quantification. Our methodology uses a novel differentiable probabilistic projection layer (DPPL) that can be combined with a wide range of neural network architectures. DPPL allows the model to learn the system in an end-to-end manner, compared to other approaches where constraints are satisfied either through a post-processing step or at inference. ProbHardE2E optimizes a strictly proper scoring rule, without making any distributional assumptions on the target, which enables it to obtain robust distributional estimates (in contrast to existing approaches that generally optimize likelihood-based objectives, which are heavily biased by their distributional assumptions and model choices); and it can incorporate a range of non-linear constraints (increasing the power of modeling and flexibility). We apply ProbHardE2E in learning partial differential equations with uncertainty estimates and to probabilistic time-series forecasting, showcasing it as a broadly applicable general framework that connects these seemingly disparate domains.

Figures (5)

• Figure 1: ProbHardE2E on the various constraint types. (a) Linear Equality: Average time per iteration (in seconds) for ProbHardE2E, compared to the HierE2E on five hierarchical time-series datasets; (b) Nonlinear Equality: Mean $\pm$3 standard deviation for the PME with conservation constraint at time $t=0.51$, with PDE parameter $m_{\text{train}}\in[3,4]$ and $m_{\text{test}}=3.88$; (c) Convex Inequality: Mean $\pm$3 standard deviation for linear advection with TVD constraint at time $t=0.51$, with PDE parameter $\beta_{\text{train}} \in [1,2]$ and $\beta_{\text{test}} = 1.5$. The horizontal axes in (b)-(c) are zoomed in to highlight the uncertainty near the propagating front.
• Figure 2: ProbHardE2E serves as a probabilistic unified framework for learning with hard constraints.
• Figure 3: Example hierarchical time series structure with $a_t \in \mathbb{R}^4$, $b_t \in \mathbb{R}^6$ and $S_{\text{sum}} = 111111100110000111011000$.
• Figure 4: Schematic representation of ProbHardE2E (see \ref{['alg:probend2end']}). Here, a known pathwise-differentiable probabilistic model is chosen to predict a (unconstrained) prior distribution. (Optionally, the projection matrix can be specified as a part of the prediction from the probabilistic model or modeled separately.) Next, we transform the distribution with our DPPL to obtain the transformed distribution, done empirically or via the Delta Method (see \ref{['subsxn:postrlocscale']}), which enforces the constraints. Lastly, we choose an appropriate loss function, e.g., CRPS, to calibrate the transformed distribution with the target variable.
• Figure 5: ProbHardE2E: PDE timing comparisons for our sampling-free approach.

Theorems & Definitions (15)

• Theorem 3.1
• Theorem B.1
• proof
• Proposition C.1
• proof
• Proposition C.2
• proof
• Proposition C.3
• proof
• Proposition D.1