Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Real-Time Machine-Learning-Based Optimization Using Input Convex Long Short-Term Memory Network

Zihao Wang, Donghan Yu, Zhe Wu

TL;DR

The paper tackles the slow, non-convex nature of neural network–based optimization for real-time control by introducing Input Convex LSTM (IC-LSTM), which enforces convexity with respect to inputs through architectural constraints and input expansion. It provides theoretical convexity guarantees and a practical implementation pathway, including a TensorFlow/Keras IC-LSTM cell and a skip-connection design, then validates the approach in two real-world case studies: a Solar PV energy system and a CSTR. Results show IC-LSTM offers substantial runtime speedups (e.g., at least 4x faster than conventional LSTM in the solar PV MPC) while maintaining acceptable closed-loop performance, though some loss in modeling fidelity due to convexification is observed. Collectively, the work demonstrates a viable route to real-time, neural-network–based optimization for energy and chemical processes, with public code to facilitate adoption and further development.

Abstract

Neural network-based optimization and control methods, often referred to as black-box approaches, are increasingly gaining attention in energy and manufacturing systems, particularly in situations where first-principles models are either unavailable or inaccurate. However, their non-convex nature significantly slows down the optimization and control processes, limiting their application in real-time decision-making processes. To address this challenge, we propose a novel Input Convex Long Short-Term Memory (IC-LSTM) network to enhance the computational efficiency of neural network-based optimization. Through two case studies employing real-time neural network-based optimization for optimizing energy and chemical systems, we demonstrate the superior performance of IC-LSTM-based optimization in terms of runtime. Specifically, in a real-time optimization problem of a real-world solar photovoltaic energy system at LHT Holdings in Singapore, IC-LSTM-based optimization achieved at least 4-fold speedup compared to conventional LSTM-based optimization. These results highlight the potential of IC-LSTM networks to significantly enhance the efficiency of neural network-based optimization and control in practical applications. Source code is available at https://github.com/killingbear999/ICLSTM.

Real-Time Machine-Learning-Based Optimization Using Input Convex Long Short-Term Memory Network

TL;DR

The paper tackles the slow, non-convex nature of neural network–based optimization for real-time control by introducing Input Convex LSTM (IC-LSTM), which enforces convexity with respect to inputs through architectural constraints and input expansion. It provides theoretical convexity guarantees and a practical implementation pathway, including a TensorFlow/Keras IC-LSTM cell and a skip-connection design, then validates the approach in two real-world case studies: a Solar PV energy system and a CSTR. Results show IC-LSTM offers substantial runtime speedups (e.g., at least 4x faster than conventional LSTM in the solar PV MPC) while maintaining acceptable closed-loop performance, though some loss in modeling fidelity due to convexification is observed. Collectively, the work demonstrates a viable route to real-time, neural-network–based optimization for energy and chemical processes, with public code to facilitate adoption and further development.

Abstract

Neural network-based optimization and control methods, often referred to as black-box approaches, are increasingly gaining attention in energy and manufacturing systems, particularly in situations where first-principles models are either unavailable or inaccurate. However, their non-convex nature significantly slows down the optimization and control processes, limiting their application in real-time decision-making processes. To address this challenge, we propose a novel Input Convex Long Short-Term Memory (IC-LSTM) network to enhance the computational efficiency of neural network-based optimization. Through two case studies employing real-time neural network-based optimization for optimizing energy and chemical systems, we demonstrate the superior performance of IC-LSTM-based optimization in terms of runtime. Specifically, in a real-time optimization problem of a real-world solar photovoltaic energy system at LHT Holdings in Singapore, IC-LSTM-based optimization achieved at least 4-fold speedup compared to conventional LSTM-based optimization. These results highlight the potential of IC-LSTM networks to significantly enhance the efficiency of neural network-based optimization and control in practical applications. Source code is available at https://github.com/killingbear999/ICLSTM.
Paper Structure (30 sections, 4 theorems, 24 equations, 11 figures, 5 tables)

This paper contains 30 sections, 4 theorems, 24 equations, 11 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Nonlinear Systems and Optimization
  3. Notation
  4. Class of Systems
  5. Neural Network-Based Optimization
  6. Family of Recurrent Neural Networks and Input Convex Neural Networks
  7. Recurrent Neural Networks
  8. Simple RNN
  9. LSTM
  10. Input Convex Neural Networks
  11. ICFNN
  12. ICRNN
  13. Input Convex Long Short-Term Memory
  14. Input Convex LSTM
  15. Convexity of IC-LSTM
  16. ...and 15 more sections

Key Result

Lemma 1

The proposed IC-LSTM cell depicted in Fig. fig_IC-LSTM_cell is convex and non-decreasing from inputs to outputs (hidden states), if all the weights, i.e., $\boldsymbol{\mathbf{W}}^{(h)}, \boldsymbol{\mathbf{W}}^{(x)}, \boldsymbol{\mathbf{D}}^{(f)}, \boldsymbol{\mathbf{D}}^{(i)},\boldsymbol{\mathbf{D

Figures (11)

  • Figure 1: System architecture of neural network-based optimization.
  • Figure 2: Architecture of IC-LSTM.
  • Figure 3: 3D plots of bivariate scalar functions, where 'true' represents the underlying non-convex function and 'pred' represents the convex form learned by IC-LSTM.
  • Figure 4: LHT Holdings technical wood production pipeline.
  • Figure 5: LHT Holdings solar PV system.
  • ...and 6 more figures

Theorems & Definitions (15)

  • Remark 1
  • Remark 2
  • Lemma 1
  • proof
  • Remark 3
  • Theorem 1
  • proof
  • Remark 4
  • Lemma 2
  • Theorem 2
  • ...and 5 more