Continuous-Time Digital Twin with Analogue Memristive Neural Ordinary Differential Equation Solver

TL;DR

A memristive neural ordinary differential equation (ODE) solver for digital twins is introduced, capable of capturing continuous-time dynamics and facilitates the modelling of complex systems using an infinite-depth model, thus enhancing both speed and energy efficiency.

Abstract

Digital twins, the cornerstone of Industry 4.0, replicate real-world entities through computer models, revolutionising fields such as manufacturing management and industrial automation. Recent advances in machine learning provide data-driven methods for developing digital twins using discrete-time data and finite-depth models on digital computers. However, this approach fails to capture the underlying continuous dynamics and struggles with modelling complex system behaviour. Additionally, the architecture of digital computers, with separate storage and processing units, necessitates frequent data transfers and Analogue-Digital (A/D) conversion, thereby significantly increasing both time and energy costs. Here, we introduce a memristive neural ordinary differential equation (ODE) solver for digital twins, which is capable of capturing continuous-time dynamics and facilitates the modelling of complex systems using an infinite-depth model. By integrating storage and computation within analogue memristor arrays, we circumvent the von Neumann bottleneck, thus enhancing both speed and energy efficiency. We experimentally validate our approach by developing a digital twin of the HP memristor, which accurately extrapolates its nonlinear dynamics, achieving a 4.2-fold projected speedup and a 41.4-fold projected decrease in energy consumption compared to state-of-the-art digital hardware, while maintaining an acceptable error margin. Additionally, we demonstrate scalability through experimentally grounded simulations of Lorenz96 dynamics, exhibiting projected performance improvements of 12.6-fold in speed and 189.7-fold in energy efficiency relative to traditional digital approaches. By harnessing the capabilities of fully analogue computing, our breakthrough accelerates the development of digital twins, offering an efficient and rapid solution to meet the demands of Industry 4.0.

Figures (4)

• Figure 1: A digital twin using a memristive neural ordinary differential equation (ODE) solver compared to a recurrent ResNet-based digital twin on conventional digital hardware in terms of data, model, and architecture. a, Background, The digital twin is a virtual counterpart of a physical asset. For example, environmental sensors capture data on atmospheric pressure, temperature, and precipitation, which is received by the digital twin. The digital twin models the dynamics of the physical system (e.g., atmospheric physics in this example). b, Data, (upper) The discretisation of continuous-time signals introduces truncation errors. (bottom) In contrast, our digital twin, using a memristive neural ODE solver, is intrinsically time-continuous. c, Model, (upper) Recurrent ResNet, by stacking neural network (NN) blocks, is a finite-depth network that parameterize a single discrete time transition using the ResNet. The representational capability scales with the depth, along with parameter population and training overhead. (bottom) In contrast, we employ a neural ODE that is equivalent to an infinite-depth neural network, featuring better performance and low training cost. d, Architecture, (upper) Conventional digital systems with frequent analogue-digital (A/D) conversion. The energy efficiency and parallelism are limited by the von Neumann bottleneck and the slowdown of Moore's Law. (bottom) In contrast, the in-memory neural ODE solver using the memristor array enables fully analogue computing without frequent data shuttling or the need for analogue-digital conversion.
• Figure 2: Hardware implementation of memristive neural ordinary differential equation (ODE) solver and the characteristics of the analogue memristor array.a, System diagram, consisting of an IVP integrator, peripheral circuit, and analogue memristor arrays. b, The IVP integrator consists of a , an integrator with analogue multiplexers, and a voltage inverter. It integrates output from the analogue memristive neural network and feeds back to the input of the analogue memristive neural network, equivalent to the differential operator of a neural ODE. c, The IVP integrator's dual modes: initial conditioning and current integration, with oscilloscope-acquired waveforms. d, The peripheral circuit provides analogue activation, current-to-voltage conversion, voltage inversion, and protection. e, The and inverter in the periphery circuit convert currents to voltages for the next layer's inputs, protected by a clamp circuit. The right panel shows oscilloscope-acquired waveform. f, The analogue memristor array uses differential pairs for weight mapping to represent both positive and negative weights. g, Optical photo of the 180nm 32$\times$32 analogue memristor array and magnified crossbar, as well as cross-sectional TEM of an analogue memristor, fabricated between the metal 4 and metal 5 layers. h, Analogue programming of individual memristors shows more than 64 states. i, Retention exceeding $10^5$ seconds for different analogue conductance. j, Conductance map for the letters ' H', ' K', and ' U' with a high device yield of 97.2%. k, Corresponding relative programming error distribution, with a variance of 4.36%.
• Figure 3: Experimental digital twin of an HP memristor.a, Schematic of the HP memristor, showing how the boundary between doped and undoped regions evolves under applied voltage. b, Framework of our system, with the neural ODE implemented on a three-layer analogue memristive neural network. c, Experimentally programmed differential conductance in the three analogue memristor arrays. d, Histogram illustrating the distribution of programmed conductance. e, Statistical analysis showing less than 2.2% error within the conductance range of 20µS to 100µS. f, Comparison of experimental voltage waveforms from our system and software ground truth upon sinusoidal and triangular input stimulation. g-h, Experimental voltage waveforms of selected $L_1$ and $L_2$ hidden neurons during inference. i, Three-dimensional, time-dependent Lissajous plot depicting the nonlinear current-voltage (I-V) relationship, induced by sinusoidal and triangular input stimuation and their corresponding states. j, Comparison of modelling errors with recurrent ResNet-based digital twin across four stimulation waveforms. k-l, Comparison of speed and energy consumption between recurrent ResNet on digital hardware, neural ODE on digital hardware, and our system across different model sizes, demonstrating projected improvements of 4.2-folder and 41.4-folder, respectively, at a hidden size of 64.
• Figure 4: Multivariate time series extrapolation of Lorenz96 dynamics with our digital twin.a, Illustration of training framework for our digital twin. b, The architecture of the digital twin, featuring fully connected layers and a group of six IVP integrators, facilitates the dynamic self-evolution of six-dimensional states ($y_1$ to $y_6$) without external stimulation. c, Temporal evolution of atmospheric pressure $y_1$ at $30^\circ$N, $0^\circ$E, modelled using Lorenz96 dynamics, depicting annual fluctuations in Earth's atmospheric pressure with detailed snapshots for March and September. d, Absolute error ($L_1$) between the output of our system and the ground truth, with the interpolation stage (0-36s) highlighted in red and the extrapolation stage (36-48s) marked in blue. e-f, Contour plots showcasing the selected interpolation (0-8s) and extrapolation (36-44s) phases, including the ground truth, output results from our system, and the $L_1$ error. g, Comparison of $L_1$ error in interpolation (red) and extrapolation (blue) for four different models: LSTM, GRU, and RNN on digital hardware and our system. h-i, Comparison of execution time and energy consumption for 5 systems (neural ODE, LSTM, GRU, and RNN on digital hardware as well as our system) executing an interpolation task from 0s to 36s, across increasing model sizes of 64, 128, 256, and 512. j, Noise analysis of our system, evaluating the impact of various combinations of read and programming noise during the inference. Our system is robust to read noise.