Application-Driven Learning: A Closed-Loop Prediction and Optimization Approach Applied to Dynamic Reserves and Demand Forecasting
Joaquim Dias Garcia, Alexandre Street, Tito Homem-de-Mello, Francisco D. Muñoz
TL;DR
The paper tackles the inefficiency of open-loop forecast–decision pipelines in power systems by introducing application-driven learning, a closed-loop, bilevel framework that co-optimizes point forecasts with downstream planning and assessment costs. It establishes convergence guarantees to the best application-aligned estimator for linear-right-hand-side problems and provides two solution paths: an exact MILP-based method via KKT conditions and a scalable, derivative-free heuristic suitable for large-scale systems. The methodology is specialized to dynamic reserves and demand forecasting, enabling joint optimization of nodal demand and up/down reserve forecasts, with a detailed case-study suite showing scalability up to 13,659 buses and substantial out-of-sample cost improvements over traditional LS-based open-loop methods. The results indicate the approach can meaningfully reduce operating costs in hour-ahead planning and offers a scientifically grounded alternative to ad hoc industry practices for reserve sizing and load biasing. Overall, the work advances theory and practice by coupling prediction and optimization in a tractable, scalable framework for large-scale power-system applications, with clear pathways for extensions to non-linear models and broader uncertainty structures.
Abstract
Forecasting and decision-making are generally modeled as two sequential steps with no feedback, following an open-loop approach. In this paper, we present application-driven learning, a new closed-loop framework in which the processes of forecasting and decision-making are merged and co-optimized through a bilevel optimization problem. We present our methodology in a general format and prove that the solution converges to the best estimator in terms of the expected cost of the selected application. Then, we propose two solution methods: an exact method based on the KKT conditions of the second-level problem and a scalable heuristic approach suitable for decomposition methods. The proposed methodology is applied to the relevant problem of defining dynamic reserve requirements and conditional load forecasts, offering an alternative approach to current ad hoc procedures implemented in industry practices. We benchmark our methodology with the standard sequential least-squares forecast and dispatch planning process. We apply the proposed methodology to an illustrative system and to a wide range of instances, from dozens of buses to large-scale realistic systems with thousands of buses. Our results show that the proposed methodology is scalable and yields consistently better performance than the standard open-loop approach.