Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-Tuning a Time Series Foundation Model with Wasserstein Loss

Andrei Chernov

TL;DR

This work replaces cross-entropy loss with Wasserstein loss and demonstrates that replacing cross-entropy loss with Wasserstein loss significantly improves point estimation.

Abstract

Inspired by recent advancements in large language models (LLMs) for Natural Language Processing (NLP), there has been a surge in research focused on developing foundational models for time series forecasting. One approach involves training LLM architectures on tokenized time series data using cross-entropy loss. Although this method has demonstrated promising results, cross-entropy loss is primarily designed for classification tasks and does not account for the distance between classes. To address this limitation, we propose using the Wasserstein loss for such architectures. To validate our approach, we fine-tuned a foundational time series model on $22$ zero-shot datasets, comparing the performance of cross-entropy loss with that of Wasserstein loss. Our results demonstrate that replacing cross-entropy loss with Wasserstein loss significantly improves point estimation.

Fine-Tuning a Time Series Foundation Model with Wasserstein Loss

TL;DR

This work replaces cross-entropy loss with Wasserstein loss and demonstrates that replacing cross-entropy loss with Wasserstein loss significantly improves point estimation.

Abstract

Inspired by recent advancements in large language models (LLMs) for Natural Language Processing (NLP), there has been a surge in research focused on developing foundational models for time series forecasting. One approach involves training LLM architectures on tokenized time series data using cross-entropy loss. Although this method has demonstrated promising results, cross-entropy loss is primarily designed for classification tasks and does not account for the distance between classes. To address this limitation, we propose using the Wasserstein loss for such architectures. To validate our approach, we fine-tuned a foundational time series model on zero-shot datasets, comparing the performance of cross-entropy loss with that of Wasserstein loss. Our results demonstrate that replacing cross-entropy loss with Wasserstein loss significantly improves point estimation.
Paper Structure (17 sections, 5 equations, 3 figures, 2 tables)

This paper contains 17 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Time Series Forecasting.
  4. Wasserstein Loss.
  5. Wasserstein Deep Learning Model for Tokenized Time Series
  6. Time Series Preprocessing
  7. Model Architecture
  8. Loss Function
  9. Forecasting and Evaluation Metrics
  10. Experiments
  11. Datasets
  12. Fine-Tuning Results
  13. Discussion
  14. Future Work
  15. Results for each dataset
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Performance Comparison: MASE
  • Figure 2: Performance Comparison: WQL
  • Figure 3: Kernel density estimation (KDE) comparison of forecasts on the FRED-MD dataset. The plot shows that the model trained with W1 loss produces significantly sharper forecast distributions compared to the model trained with cross-entropy loss.