Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge-enhanced Transformer for Multivariate Long Sequence Time-series Forecasting

Shubham Tanaji Kakde, Rony Mitra, Jasashwi Mandal, Manoj Kumar Tiwari

TL;DR

A novel approach is introduced that encapsulates conceptual relationships among variables within a well-defined knowledge graph, forming dynamic and learnable KGEs for seamless integration into the transformer architecture, which improves the accuracy of multivariate LSTF by capturing complex temporal and relational dynamics across multiple domains.

Abstract

Multivariate Long Sequence Time-series Forecasting (LSTF) has been a critical task across various real-world applications. Recent advancements focus on the application of transformer architectures attributable to their ability to capture temporal patterns effectively over extended periods. However, these approaches often overlook the inherent relationships and interactions between the input variables that could be drawn from their characteristic properties. In this paper, we aim to bridge this gap by integrating information-rich Knowledge Graph Embeddings (KGE) with state-of-the-art transformer-based architectures. We introduce a novel approach that encapsulates conceptual relationships among variables within a well-defined knowledge graph, forming dynamic and learnable KGEs for seamless integration into the transformer architecture. We investigate the influence of this integration into seminal architectures such as PatchTST, Autoformer, Informer, and Vanilla Transformer. Furthermore, we thoroughly investigate the performance of these knowledge-enhanced architectures along with their original implementations for long forecasting horizons and demonstrate significant improvement in the benchmark results. This enhancement empowers transformer-based architectures to address the inherent structural relation between variables. Our knowledge-enhanced approach improves the accuracy of multivariate LSTF by capturing complex temporal and relational dynamics across multiple domains. To substantiate the validity of our model, we conduct comprehensive experiments using Weather and Electric Transformer Temperature (ETT) datasets.

Knowledge-enhanced Transformer for Multivariate Long Sequence Time-series Forecasting

TL;DR

A novel approach is introduced that encapsulates conceptual relationships among variables within a well-defined knowledge graph, forming dynamic and learnable KGEs for seamless integration into the transformer architecture, which improves the accuracy of multivariate LSTF by capturing complex temporal and relational dynamics across multiple domains.

Abstract

Multivariate Long Sequence Time-series Forecasting (LSTF) has been a critical task across various real-world applications. Recent advancements focus on the application of transformer architectures attributable to their ability to capture temporal patterns effectively over extended periods. However, these approaches often overlook the inherent relationships and interactions between the input variables that could be drawn from their characteristic properties. In this paper, we aim to bridge this gap by integrating information-rich Knowledge Graph Embeddings (KGE) with state-of-the-art transformer-based architectures. We introduce a novel approach that encapsulates conceptual relationships among variables within a well-defined knowledge graph, forming dynamic and learnable KGEs for seamless integration into the transformer architecture. We investigate the influence of this integration into seminal architectures such as PatchTST, Autoformer, Informer, and Vanilla Transformer. Furthermore, we thoroughly investigate the performance of these knowledge-enhanced architectures along with their original implementations for long forecasting horizons and demonstrate significant improvement in the benchmark results. This enhancement empowers transformer-based architectures to address the inherent structural relation between variables. Our knowledge-enhanced approach improves the accuracy of multivariate LSTF by capturing complex temporal and relational dynamics across multiple domains. To substantiate the validity of our model, we conduct comprehensive experiments using Weather and Electric Transformer Temperature (ETT) datasets.

Paper Structure

This paper contains 16 sections, 7 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Methodology
  4. Model Structure
  5. Input Embeddings
  6. Multi-Head Attention in Encoder-Decoder
  7. Loss Function
  8. Construction of Dynamic and Learnable Knowledge Graph Embeddings from Knowledge Graph
  9. Experiments
  10. Datasets
  11. Weather
  12. ETT
  13. Knowledge Graphs for benchmark datasets
  14. Experimental Setup
  15. Results and Comparison
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Venn diagram illustrating the central role of Knowledge Graph Embeddings at the intersection of Knowledge Graphs, Long Sequence Time-series Forecasting, and Transformer architectures
  • Figure 2: Model structure of knowledge-enhanced transformer for LSTF; Knowledge graph embeddings along with Positional embeddings, Value embeddings and Temporal embedding are added with input embeddings towards encoder and decoder blocks
  • Figure 3: Flowchart for construction of learnable Knowledge Graph Embeddings (KGE)
  • Figure 4: Knowledge Graph for Weather dataset; variable relationships are derived from theoretical laws and established empirical studies