TS-Haystack: A Multi-Scale Retrieval Benchmark for Time Series Language Models

TL;DR

This work addresses the challenge of long-context retrieval in Time Series Language Models (TSLMs) by introducing TS-Haystack, a long-context retrieval benchmark built from real accelerometer data with needle-insertion events. It systematically evaluates classification versus retrieval across multiple context lengths using two architectures and an oracle variant, revealing a task-dependent trade-off: latent compression can improve classification up to a compression of $176\times$ while retrieval quality degrades with longer contexts, and even plateaus rather than decays monotonically. The results indicate the LLM backbone has sufficient reasoning capacity, but the bottleneck lies in the time-series encoder and projection layers, motivating architectural designs that decouple sequence length from computational cost while preserving temporal fidelity. The TS-Haystack benchmark thus provides a diagnostic tool to guide future long-context designs for real-world, extended-time-series understanding and retrieval tasks.

Abstract

Time Series Language Models (TSLMs) are emerging as unified models for reasoning over continuous signals in natural language. However, long-context retrieval remains a major limitation: existing models are typically trained and evaluated on short sequences, while real-world time-series sensor streams can span millions of datapoints. This mismatch requires precise temporal localization under strict computational constraints, a regime that is not captured by current benchmarks. We introduce TS-Haystack, a long-context temporal retrieval benchmark comprising ten task types across four categories: direct retrieval, temporal reasoning, multi-step reasoning and contextual anomaly. The benchmark uses controlled needle insertion by embedding short activity bouts into longer longitudinal accelerometer recordings, enabling systematic evaluation across context lengths ranging from seconds to 2 hours per sample. We hypothesize that existing TSLM time series encoders overlook temporal granularity as context length increases, creating a task-dependent effect: compression aids classification but impairs retrieval of localized events. Across multiple model and encoding strategies, we observe a consistent divergence between classification and retrieval behavior. Learned latent compression preserves or improves classification accuracy at compression ratios up to 176$\times$, but retrieval performance degrades with context length, incurring in the loss of temporally localized information. These results highlight the importance of architectural designs that decouple sequence length from computational complexity while preserving temporal fidelity.

Figures (12)

• Figure 1: TS-Haystack construction pipeline.Left: A 1 hour window is sampled from a participant, activity needles ranging 1s -- 6s are sampled at random. Right: Question-Answer pairs and chain-of-thought rationales are generated from plots and metadata-enriched prompt templates.
• Figure 2: Scaling behaviors. (a) Capture24 classification: Macro-F1 and Balanced Accuracy increase with context length. (b) TS-Haystack: Retrieval accuracy degrades with context length.
• Figure 3: Task 1: Existence. The model must detect whether a specific activity is present among same-regime distractors.
• Figure 4: Task 2 (Localization). The model must identify and temporally ground a specific activity
• Figure 5: Task 3 (Counting). The model must enumerate all occurrences of a specific activity