Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lifting Manifolds to Mitigate Pseudo-Alignment in LLM4TS

Liangwei Nathan Zheng, Wenhao Liang, Wei Emma Zhang, Miao Xu, Olaf Maennel, Weitong Chen

TL;DR

The paper tackles pseudo-alignment in LLM-based time-series forecasting by linking it to the cone effect in pretrained LLMs and the inherently low-dimensional manifold of time-series data. It introduces TimeSUP, a Time Manifold Enhancer that patches time-series inputs, selects Top-K language prototypes, and fuses time-text tokens with two lightweight MLPs to significantly raise the time-series manifold dimensionality toward the language embedding space. Empirical results across eight long-horizon datasets show TimeSUP improves forecasting performance and mitigates pseudo-alignment, and the method is readily pluggable into multiple LLM4TS pipelines. The work provides both a theoretical framing and a practical, broadly applicable solution that enhances cross-modal learning without sacrificing modality-specific structure.

Abstract

Pseudo-Alignment is a pervasive challenge in many large language models for time series (LLM4TS) models, often causing them to underperform compared to linear models or randomly initialised backbones. However, there is limited discussion in the community for the reasons that pseudo-alignment occurs. In this work, we conduct a thorough investigation into the root causes of pseudo-alignment in LLM4TS and build a connection of pseudo-alignment to the cone effect in LLM. We demonstrate that pseudo-alignment arises from the interplay of cone effect within pretrained LLM components and the intrinsically low-dimensional manifold of time-series data. In addition, we also introduce \textit{\textbf{TimeSUP}}, a novel technique designed to mitigate this issue and improve forecast performance in existing LLM4TS approaches. TimeSUP addresses this by increasing the time series manifold to more closely match the intrinsic dimension of language embeddings, allowing the model to distinguish temporal signals clearly while still capturing shared structures across modalities. As a result, representations for time and language tokens remain distinct yet exhibit high cosine similarity, signifying that the model preserves each modality unique features while learning their commonalities in a unified embedding space. Empirically, TimeSUP consistently outperforms state-of-the-art LLM4TS methods and other lightweight baselines on long-term forecasting performance. Furthermore, it can be seamlessly integrated into four existing LLM4TS pipelines and delivers significant improvements in forecasting performance.

Lifting Manifolds to Mitigate Pseudo-Alignment in LLM4TS

TL;DR

The paper tackles pseudo-alignment in LLM-based time-series forecasting by linking it to the cone effect in pretrained LLMs and the inherently low-dimensional manifold of time-series data. It introduces TimeSUP, a Time Manifold Enhancer that patches time-series inputs, selects Top-K language prototypes, and fuses time-text tokens with two lightweight MLPs to significantly raise the time-series manifold dimensionality toward the language embedding space. Empirical results across eight long-horizon datasets show TimeSUP improves forecasting performance and mitigates pseudo-alignment, and the method is readily pluggable into multiple LLM4TS pipelines. The work provides both a theoretical framing and a practical, broadly applicable solution that enhances cross-modal learning without sacrificing modality-specific structure.

Abstract

Pseudo-Alignment is a pervasive challenge in many large language models for time series (LLM4TS) models, often causing them to underperform compared to linear models or randomly initialised backbones. However, there is limited discussion in the community for the reasons that pseudo-alignment occurs. In this work, we conduct a thorough investigation into the root causes of pseudo-alignment in LLM4TS and build a connection of pseudo-alignment to the cone effect in LLM. We demonstrate that pseudo-alignment arises from the interplay of cone effect within pretrained LLM components and the intrinsically low-dimensional manifold of time-series data. In addition, we also introduce \textit{\textbf{TimeSUP}}, a novel technique designed to mitigate this issue and improve forecast performance in existing LLM4TS approaches. TimeSUP addresses this by increasing the time series manifold to more closely match the intrinsic dimension of language embeddings, allowing the model to distinguish temporal signals clearly while still capturing shared structures across modalities. As a result, representations for time and language tokens remain distinct yet exhibit high cosine similarity, signifying that the model preserves each modality unique features while learning their commonalities in a unified embedding space. Empirically, TimeSUP consistently outperforms state-of-the-art LLM4TS methods and other lightweight baselines on long-term forecasting performance. Furthermore, it can be seamlessly integrated into four existing LLM4TS pipelines and delivers significant improvements in forecasting performance.
Paper Structure (16 sections, 1 theorem, 9 equations, 5 figures, 5 tables)

This paper contains 16 sections, 1 theorem, 9 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Time Series Lies in a Lower-Dimensional Data Manifold
  4. Cone Effect in Large Language Models
  5. Understand Pseudo-Alignment
  6. Address Pseudo-Alignment
  7. Related Work
  8. Lightweight Time Series Models
  9. Large Language Models for Time Series
  10. Experiment
  11. Primary Results
  12. Pseudo-Alignment in LLM Layers
  13. Is the Enhancer Useful for other LLM4TS Model?
  14. What Components in LLM Manipulate the Feature Map of Time Series?
  15. Conclusion
  16. ...and 1 more sections

Key Result

Theorem 1

When the manifold dimensionality $m \to 0$ and $n \to 0$, The cosine similarity tends to converge to the similarity between the mean of time series and language distribution only without the instance-specific feature involved, leading to pseudo-alignment. Formally, we can represent this pseudo-align

Figures (5)

  • Figure 1: PCA Explained Variance Ratio plot for Language, Original, and Enhanced Time series
  • Figure 2: Comparison between the representation and similarity with and without TimeSUP: (1) Cosine similarity is determined by paring time series and language tokens. (2) Pseudo-Alignment is still preserved if time series token is fed in LLM. (3) Pseudo-Alignment phenomenon is mitigated by TimeSUP.
  • Figure 3: Overall Framework of Time Manifold Enhancer to address Pseudo-Alignment: (1) Text prototype is generated by a linear combination on the vocabulary and a one layer multihead attention is used to find text prototype best describing time series patches. (2) Each time patch will pick a top-k text prototype as the material to enhance dimension of time manifold. (3) Two MLP mixer are applied over the token dimension and feature dimension to compress and fuse the time-text tokens and generate the enhanced version of time tokens. (4) All fused time-text tokens will be sent to LLM for final prediction.
  • Figure 4: Representation in Layers: the cosine similarity is calculated between time and language tokens after each layer, Blue is language and Orange is time series. LLM starts to recognize time series token after layer 2 as two modalities are separable while maintain high cosine similarity.
  • Figure 5: UMAP Representation for Different Setting on ETTh1-720 dataset. Blue is language and Orange is time series. Notation: LayerNorm(LN), MultiHead Attention(MHA), Pretrained–Frozen (PF), Pretrained–fineTuned (PT), Random–Frozen (RF), and Random–fineTuned (RT).

Theorems & Definitions (1)

  • Theorem 1