Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tri-Level Navigator: LLM-Empowered Tri-Level Learning for Time Series OOD Generalization

Chengtao Jian, Kai Yang, Yang Jiao

TL;DR

A novel novel TTSO framework for time series OOD generalization via pre-trained Large Language Models, and a stratified localization algorithm tailored for this tri-level optimization problem, theoretically demonstrating the guaranteed convergence of the proposed algorithm.

Abstract

Out-of-Distribution (OOD) generalization in machine learning is a burgeoning area of study. Its primary goal is to enhance the adaptability and resilience of machine learning models when faced with new, unseen, and potentially adversarial data that significantly diverges from their original training datasets. In this paper, we investigate time series OOD generalization via pre-trained Large Language Models (LLMs). We first propose a novel \textbf{T}ri-level learning framework for \textbf{T}ime \textbf{S}eries \textbf{O}OD generalization, termed TTSO, which considers both sample-level and group-level uncertainties. This formula offers a fresh theoretic perspective for formulating and analyzing OOD generalization problem. In addition, we provide a theoretical analysis to justify this method is well motivated. We then develop a stratified localization algorithm tailored for this tri-level optimization problem, theoretically demonstrating the guaranteed convergence of the proposed algorithm. Our analysis also reveals that the iteration complexity to obtain an $ε$-stationary point is bounded by O($\frac{1}{ε^{2}}$). Extensive experiments on real-world datasets have been conducted to elucidate the effectiveness of the proposed method.

Tri-Level Navigator: LLM-Empowered Tri-Level Learning for Time Series OOD Generalization

TL;DR

A novel novel TTSO framework for time series OOD generalization via pre-trained Large Language Models, and a stratified localization algorithm tailored for this tri-level optimization problem, theoretically demonstrating the guaranteed convergence of the proposed algorithm.

Abstract

Out-of-Distribution (OOD) generalization in machine learning is a burgeoning area of study. Its primary goal is to enhance the adaptability and resilience of machine learning models when faced with new, unseen, and potentially adversarial data that significantly diverges from their original training datasets. In this paper, we investigate time series OOD generalization via pre-trained Large Language Models (LLMs). We first propose a novel \textbf{T}ri-level learning framework for \textbf{T}ime \textbf{S}eries \textbf{O}OD generalization, termed TTSO, which considers both sample-level and group-level uncertainties. This formula offers a fresh theoretic perspective for formulating and analyzing OOD generalization problem. In addition, we provide a theoretical analysis to justify this method is well motivated. We then develop a stratified localization algorithm tailored for this tri-level optimization problem, theoretically demonstrating the guaranteed convergence of the proposed algorithm. Our analysis also reveals that the iteration complexity to obtain an -stationary point is bounded by O(). Extensive experiments on real-world datasets have been conducted to elucidate the effectiveness of the proposed method.

Paper Structure

This paper contains 34 sections, 8 theorems, 54 equations, 6 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation and Algorithm
  4. Preliminary
  5. A Tri-level Learning Framework
  6. Stratified Localization Algorithm
  7. TTSO for Fine-tuning LLMs
  8. Convergence Analysis
  9. Experiment
  10. Main Results
  11. Ablation Study
  12. Conclusion
  13. Acknowledgements
  14. Theoretical Proofs and Discussion
  15. Proof of Proposition \ref{['eq:valid_cp']}
  16. ...and 19 more sections

Key Result

Theorem 1

For any two augmentation methods $a, a^{\prime} \in \mathcal{A}$, representation function $r_\theta$ and classifier $h_\omega$, we have Fix $r_\theta$, let $h_a={\arg \min }_{h_\omega} \mathcal{R}\left(h_\omega \circ r ; \mathbb{P}_{\text{train }}\right)$, we have

Figures (6)

  • Figure 1: The depiction of sample-level, group-level, and combined uncertainties.
  • Figure 2: Ablation study of TTSO$^{*}$
  • Figure 3: Structure of LLM Fine-tuning with TTSO, illustrating the two-phase approach starting with alignment fine-tuning followed by downstream fine-tuning, adapted specifically for time series out-of-distribution generalization tasks.
  • Figure 4: Sample-Level Uncertainty: Each line represents a window of time series data with the same label.
  • Figure 5: Group-Level Uncertainty: Histogram of 'x' axis values, with each color representing a different group.
  • ...and 1 more figures

Theorems & Definitions (11)

  • Definition 1: Invariant Risk Minimization arjovsky2019invariant
  • Definition 2: Augmentation Robust Alignment Loss zhao2022arcl
  • Theorem 1: Upper Bound of Risk Gap Between Augmented Domains shen2021oodg
  • Theorem 2: Upper Bound on Target Error
  • Theorem 3
  • Proposition 1
  • Definition 3: $\epsilon$-Stationary Point
  • Theorem 4: Convergence Guarantee
  • Theorem 5: Convergence Rate
  • Theorem 6
  • ...and 1 more