Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Physics-inspired Energy Transition Neural Network for Sequence Learning

Zhou Wu, Junyi An, Baile Xu, Furao Shen, Jian Zhao

TL;DR

PETNN addresses long-term dependency challenges in sequence learning by grounding a recurrent architecture in physics energy transition concepts. It introduces the Remaining Time $T_t$, Cell State $C_t$, and Hidden State $S_t$, with updates driven by energy injection and a self-selective information mixing mechanism. Empirical results on time-series forecasting and text sentiment classification show PETNN competitive with Transformer-based models and often superior, while requiring lower computational resources due to its recurrence. This physics-inspired approach offers a memory-centric, generalizable foundation for future recurrence-based architectures beyond Transformer-dominated regimes.

Abstract

Recently, the superior performance of Transformers has made them a more robust and scalable solution for sequence modeling than traditional recurrent neural networks (RNNs). However, the effectiveness of Transformer in capturing long-term dependencies is primarily attributed to their comprehensive pair-modeling process rather than inherent inductive biases toward sequence semantics. In this study, we explore the capabilities of pure RNNs and reassess their long-term learning mechanisms. Inspired by the physics energy transition models that track energy changes over time, we propose a effective recurrent structure called the``Physics-inspired Energy Transition Neural Network" (PETNN). We demonstrate that PETNN's memory mechanism effectively stores information over long-term dependencies. Experimental results indicate that PETNN outperforms transformer-based methods across various sequence tasks. Furthermore, owing to its recurrent nature, PETNN exhibits significantly lower complexity. Our study presents an optimal foundational recurrent architecture and highlights the potential for developing effective recurrent neural networks in fields currently dominated by Transformer.

Physics-inspired Energy Transition Neural Network for Sequence Learning

TL;DR

PETNN addresses long-term dependency challenges in sequence learning by grounding a recurrent architecture in physics energy transition concepts. It introduces the Remaining Time , Cell State , and Hidden State , with updates driven by energy injection and a self-selective information mixing mechanism. Empirical results on time-series forecasting and text sentiment classification show PETNN competitive with Transformer-based models and often superior, while requiring lower computational resources due to its recurrence. This physics-inspired approach offers a memory-centric, generalizable foundation for future recurrence-based architectures beyond Transformer-dominated regimes.

Abstract

Recently, the superior performance of Transformers has made them a more robust and scalable solution for sequence modeling than traditional recurrent neural networks (RNNs). However, the effectiveness of Transformer in capturing long-term dependencies is primarily attributed to their comprehensive pair-modeling process rather than inherent inductive biases toward sequence semantics. In this study, we explore the capabilities of pure RNNs and reassess their long-term learning mechanisms. Inspired by the physics energy transition models that track energy changes over time, we propose a effective recurrent structure called the``Physics-inspired Energy Transition Neural Network" (PETNN). We demonstrate that PETNN's memory mechanism effectively stores information over long-term dependencies. Experimental results indicate that PETNN outperforms transformer-based methods across various sequence tasks. Furthermore, owing to its recurrent nature, PETNN exhibits significantly lower complexity. Our study presents an optimal foundational recurrent architecture and highlights the potential for developing effective recurrent neural networks in fields currently dominated by Transformer.
Paper Structure (31 sections, 19 equations, 6 figures, 10 tables, 1 algorithm)

This paper contains 31 sections, 19 equations, 6 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Model
  4. Physics-inspired Energy Transition Neuron
  5. Update process in PETNN
  6. Memory Cell
  7. Remaining Time
  8. Cell State
  9. Hidden State
  10. Physics-Inspired Design Rationale
  11. Energy Transition as a State Update Rule
  12. Time Decay as a Relaxation Process
  13. Related Work
  14. Experiments
  15. Time Series Forecasting Task
  16. ...and 16 more sections

Figures (6)

  • Figure 1: Quantum theory explains the energy transition model through the processes of absorption and release.
  • Figure 2: A basic neuron in PETNN. The black waveform represents stored neuronal information over time. When the signal reaches the red dashed line, the neuron releases this information, returning to ground state.
  • Figure 3: Internal architecture of PETNN.
  • Figure 4: Comparision of Different Update Methods.
  • Figure 5: Visualization of ETTm2 prediction by different models under the input-96-predict-336 setting.The blue lines stand for the ground truth and the orange lines stand for the prediction.
  • ...and 1 more figures