TempoGPT: Enhancing Time Series Reasoning via Quantizing Embedding

TL;DR

TempoGPT addresses the difficulty of enabling time series reasoning in multi-modal LLMs by introducing quantized temporal embeddings and a white-box data generation pipeline. It constructs a multimodal electrical time series dataset and a TempoGPT model that tokenizes temporal patches via a fixed codebook, processed by a shared embedding layer with textual tokens, enabling consistent representations. Empirical results show TempoGPT achieves state-of-the-art performance on constructed time series reasoning tasks and improves logical reasoning metrics (LRA, DR) compared to continuous-embedding baselines, validating the importance of discrete temporal tokens for multi-modal alignment. The work advances time series reasoning with practical data generation and demonstrates the benefits of quantization for TLMs in complex tasks, with potential impact on finance, engineering, and beyond.

Abstract

Multi-modal language model has made advanced progress in vision and audio, but still faces significant challenges in dealing with complex reasoning tasks in the time series domain. The reasons are twofold. First, labels for multi-modal time series data are coarse and devoid of analysis or reasoning processes. Training with these data cannot improve the model's reasoning capabilities. Second, due to the lack of precise tokenization in processing time series, the representation patterns for temporal and textual information are inconsistent, which hampers the effectiveness of multi-modal alignment. To address these challenges, we propose a multi-modal time series data construction approach and a multi-modal time series language model (TLM), TempoGPT. Specially, we construct multi-modal data for complex reasoning tasks by analyzing the variable-system relationships within a white-box system. Additionally, proposed TempoGPT achieves consistent representation between temporal and textual information by quantizing temporal embeddings, where temporal embeddings are quantized into a series of discrete tokens using a predefined codebook; subsequently, a shared embedding layer processes both temporal and textual tokens. Extensive experiments demonstrate that TempoGPT accurately perceives temporal information, logically infers conclusions, and achieves state-of-the-art in the constructed complex time series reasoning tasks. Moreover, we quantitatively demonstrate the effectiveness of quantizing temporal embeddings in enhancing multi-modal alignment and the reasoning capabilities of TLMs. Code and data are available at https://github.com/zhanghaochuan20/TempoGPT.

Figures (7)

• Figure 1: (a) TLMs perform well in tasks related to trend (Tr.) but poorly in tasks involving complex reasoning (Re.). (b) as the model size increases, the performance of TLMs deteriorates on reasoning tasks.
• Figure 2: On the left side of the diagram, due to the lack of temporal tokenizers, TLMs process inputs with inconsistent representation patterns. On the right side of the diagram, proposed TempoGPT quantizes temporal embeddings with a predefined codebook, ensuring a precise tokenization and consistent representation pattern.
• Figure 3: At the top of the diagram, the left side presents the pre-training data components, which include anomaly and temporal information, while the right side presents the electrical simulation model responsible for generating the multi-modal data. On the bottom of the diagram, the construction of fine-tuned data is showcased, primarily comprising two trend-related tasks: trend analysis and trend forecasting; along with three reasoning-related tasks: fault judgment, fault diagnosis, and fault analysis. Due to space limitations, we present detailed data in the Appendix \ref{['apx: data']}.
• Figure 4: TempoGPT network architecture. TempoGPT employs quantization encoding to tokenize time series into temporal tokens. A shared embedding layer processes these tokens alongside textual tokens to achieve consistent representation pattern and generate corresponding embeddings. These embeddings are then processed by LLMs to generate the final text response.
• Figure 5: Quiz showcases of TempoGPT (GPT-2) and Baseline (GPT-2 based on linear) in the time series reasoning tasks. Responses highlighted in red and blue indicate the true labels and incorrect perception or reasoning, respectively. For more response examples from TempoGPT, please refer to the Appendix \ref{['apx: data']}.