Beyond LoRA: Exploring Efficient Fine-Tuning Techniques for Time Series Foundational Models

TL;DR

Time Series Foundational Models struggle to adapt to healthcare due to data scarcity and domain shift. The authors benchmark two selective (BitFit, LayerNorm Tuning) and two additive (VeRA, FourierFT) PEFT techniques on the Chronos TSFM to forecast ICU vitals for sepsis patients using the public eICU dataset. They compare against LoRA and fully fine-tuned baselines, showing that several PEFT methods achieve competitive or superior performance while tuning orders of magnitude fewer parameters; FourierFT on Chronos Tiny even surpasses a state-of-the-art scratch model with only ~2.4k tunable parameters. The work underscores the viability of parameter-efficient domain adaptation for healthcare TSFMs and motivates future hybrids and scaling to larger Chronos variants.

Abstract

Time Series Foundation Models (TSFMs) have recently garnered attention for their ability to model complex, large-scale time series data across domains such as retail, finance, and transportation. However, their application to sensitive, domain-specific fields like healthcare remains challenging, primarily due to the difficulty of fine-tuning these models for specialized, out-of-domain tasks with scarce publicly available datasets. In this work, we explore the use of Parameter-Efficient Fine-Tuning (PEFT) techniques to address these limitations, focusing on healthcare applications, particularly ICU vitals forecasting for sepsis patients. We introduce and evaluate two selective (BitFit and LayerNorm Tuning) and two additive (VeRA and FourierFT) PEFT techniques on multiple configurations of the Chronos TSFM for forecasting vital signs of sepsis patients. Our comparative analysis demonstrates that some of these PEFT methods outperform LoRA in terms of parameter efficiency and domain adaptation, establishing state-of-the-art (SOTA) results in ICU vital forecasting tasks. Interestingly, FourierFT applied to the Chronos (Tiny) variant surpasses the SOTA model while fine-tuning only 2,400 parameters compared to the 700K parameters of the benchmark.