Billions-Scale Forecast Reconciliation
Tianyu, Wang, Matthew C. Johnson, Steven Klee, Matthew L. Malloy
TL;DR
Billions-Scale Forecast Reconciliation tackles coherence across extensive retail taxonomies by solving a constrained quadratic program $\min_y \frac{1}{2}\|y-\hat{y}\|_{2,\mathbf{W}}^{2}$ s.t. $\mathbf{A}y=\mathbf{0}$, $y\ge0$, where $\mathbf{A}$ encodes aggregations and $\mathbf{W}$ weights. It introduces and benchmarks three scalable solvers—Alternating Projections, Dykstra's Algorithm, and ADMM—capable of handling matrices with billions of columns, demonstrating practical feasibility beyond prior scales. The paper shows that the unconstrained LS solution $\mathbf{y}^*_{LSQR}=\hat{\mathbf{y}}-\mathbf{W}^{-1}\mathbf{A}^{\top}(\mathbf{A}\mathbf{W}^{-1}\mathbf{A}^{\top})^{-1}\mathbf{A}\hat{\mathbf{y}}$ can reproduce share-based reconciliation under specific weighting, linking optimization to classical approaches. It proves that under tree-based hierarchies with top-heavy or bottom-heavy weights, the limiting LS solutions converge to top-down or bottom-up reconciliations, indicating nonnegativity constraints can be redundant. Empirically, it validates near-optimal reconciliation on datasets with billions of forecast entries, deriving the constraint matrix from tabular data and achieving scalable performance on high-memory cloud infrastructure. The work enables scalable coherent forecasting for enterprise-scale operations and clarifies how weighting schemes govern reconciliation behavior.
Abstract
The problem of combining multiple forecasts of related quantities that obey expected equality and additivity constraints, often referred to a hierarchical forecast reconciliation, is naturally stated as a simple optimization problem. In this paper we explore optimization-based point forecast reconciliation at scales faced by large retailers. We implement and benchmark several algorithms to solve the forecast reconciliation problem, showing efficacy when the dimension of the problem exceeds four billion forecasted values. To the best of our knowledge, this is the largest forecast reconciliation problem, and perhaps on-par with the largest constrained least-squares-problem ever solved. We also make several theoretical contributions. We show that for a restricted class of problems and when the loss function is weighted appropriately, least-squares forecast reconciliation is equivalent to share-based forecast reconciliation. This formalizes how the optimization based approach can be thought of as a generalization of share-based reconciliation, applicable to multiple, overlapping data hierarchies.