Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

User-friendly Foundation Model Adapters for Multivariate Time Series Classification

Vasilii Feofanov, Romain Ilbert, Malik Tiomoko, Themis Palpanas, Ievgen Redko

TL;DR

This work tackles the high resource demands of foundation models for multivariate time series by inserting dimensionality-reduction adapters ahead of pre-trained TSFMs. It systematically compares PCA, SVD, Rand Proj, VAR, and lcomb adapters on MOMENT and ViT across twelve UEA datasets, under a strict single-GPU, two-hour budget. The results show up to 10x speedups and up to 4.5x more datasets can be processed on a single GPU, with accuracy preserved and no statistically significant differences between adapter methods. The findings demonstrate that lightweight adapters can dramatically improve the practicality and scalability of foundation models for multivariate time series classification, with PCA often delivering strongest performance and lcomb expanding dataset coverage.

Abstract

Foundation models, while highly effective, are often resource-intensive, requiring substantial inference time and memory. This paper addresses the challenge of making these models more accessible with limited computational resources by exploring dimensionality reduction techniques. Our goal is to enable users to run large pre-trained foundation models on standard GPUs without sacrificing performance. We investigate classical methods such as Principal Component Analysis alongside neural network-based adapters, aiming to reduce the dimensionality of multivariate time series data while preserving key features. Our experiments show up to a 10x speedup compared to the baseline model, without performance degradation, and enable up to 4.5x more datasets to fit on a single GPU, paving the way for more user-friendly and scalable foundation models.

User-friendly Foundation Model Adapters for Multivariate Time Series Classification

TL;DR

This work tackles the high resource demands of foundation models for multivariate time series by inserting dimensionality-reduction adapters ahead of pre-trained TSFMs. It systematically compares PCA, SVD, Rand Proj, VAR, and lcomb adapters on MOMENT and ViT across twelve UEA datasets, under a strict single-GPU, two-hour budget. The results show up to 10x speedups and up to 4.5x more datasets can be processed on a single GPU, with accuracy preserved and no statistically significant differences between adapter methods. The findings demonstrate that lightweight adapters can dramatically improve the practicality and scalability of foundation models for multivariate time series classification, with PCA often delivering strongest performance and lcomb expanding dataset coverage.

Abstract

Foundation models, while highly effective, are often resource-intensive, requiring substantial inference time and memory. This paper addresses the challenge of making these models more accessible with limited computational resources by exploring dimensionality reduction techniques. Our goal is to enable users to run large pre-trained foundation models on standard GPUs without sacrificing performance. We investigate classical methods such as Principal Component Analysis alongside neural network-based adapters, aiming to reduce the dimensionality of multivariate time series data while preserving key features. Our experiments show up to a 10x speedup compared to the baseline model, without performance degradation, and enable up to 4.5x more datasets to fit on a single GPU, paving the way for more user-friendly and scalable foundation models.
Paper Structure (28 sections, 6 figures, 5 tables)

This paper contains 28 sections, 6 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Framework
  4. Problem setup
  5. Notations.
  6. Datasets.
  7. Experimental Setup.
  8. Foundation Models.
  9. Objective.
  10. Motivation
  11. Feature-Level Transformation Methods
  12. Principal Component Analysis (PCA)
  13. Truncated Singular Value Decomposition (SVD)
  14. Random Projection (Rand Proj)
  15. Variance-Based Feature Selection (VAR)
  16. ...and 13 more sections

Figures (6)

  • Figure 1: Comparison of running times for MOMENT and ViT Foundation Models averaged across all datasets and three different seeds
  • Figure 2: Comparison of PCA and PatchPCA Methods for ViT and MOMENT Models
  • Figure 3: Performance Comparison Between lcomb and lcomb_top_k Fine-Tuning Configurations for both MOMENT and ViT Models
  • Figure 4: Comparison of Adapter's Average Rank for MOMENT and ViT Foundation Models averaged across all datasets and three different seeds
  • Figure 5: Heatmap of Pairwise p-values for Adapter Methods for MOMENT and ViT Foundation Models averaged across all datasets and three different seeds
  • ...and 1 more figures