Prompting Underestimates LLM Capability for Time Series Classification

TL;DR

This paper reevaluates large language models (LLMs) for time-series classification by contrasting prompt-based outputs with linear probes of hidden representations. It demonstrates that prompts dramatically underestimate an LLM's temporal understanding, while simple linear probes reveal strong, early-layer discriminative signals that can rival state-of-the-art time-series models, especially when visual or multimodal inputs are used. The findings hold across diverse architectures, including text-only LLMs, and suggest reframing LLMs as robust feature extractors for non-linguistic data, rather than solely reasoning agents accessed via prompts. Overall, the work highlights a systematic mismatch between prompt-based evaluation and the models' true representational capacity, urging more diagnostics that separate prompting interfaces from internal representations.

Abstract

Prompt-based evaluations suggest that large language models (LLMs) perform poorly on time series classification, raising doubts about whether they encode meaningful temporal structure. We show that this conclusion reflects limitations of prompt-based generation rather than the model's representational capacity by directly comparing prompt outputs with linear probes over the same internal representations. While zero-shot prompting performs near chance, linear probes improve average F1 from 0.15-0.26 to 0.61-0.67, often matching or exceeding specialized time series models. Layer-wise analyses further show that class-discriminative time series information emerges in early transformer layers and is amplified by visual and multimodal inputs. Together, these results demonstrate a systematic mismatch between what LLMs internally represent and what prompt-based evaluation reveals, leading current evaluations to underestimate their time series understanding.

Figures (15)

• Figure 1: Does an LLM's observed ability for time series classification match its underlying potential?
• Figure 2: Overview of time series representations and prediction paradigms. Raw time series are transformed into digit-based text representations and visualizations. These representations are incorporated into prompts for direct prediction or used to extract hidden representations for layer-wise probing with linear classifiers.
• Figure 3: Layer-wise macro F1 of linear probes for Llama versus a random baseline under digit-only (d) and multimodal (d+v) inputs, showing early emergence of discriminative time-series information.
• Figure 4: Layer-wise probe performance for Mistral and Qwen across digit (d), visual (v), and multimodal (d+v) inputs, with visual and multimodal representations yielding stronger separability.
• Figure 5: t-SNE visualizations of probe embeddings for Llama (left) and a time-series baseline (Moment, right) on the HAD dataset, illustrating comparable class separation.