Time Series, Vision, and Language: Exploring the Limits of Alignment in Contrastive Representation Spaces

TL;DR

This investigation reveals that overall alignment in contrastive representation spaces improves with model size, but this alignment is asymmetric: time series align more strongly with visual representations than with text, and images can act as effective intermediaries between time series and language.

Abstract

The Platonic Representation Hypothesis posits that learned representations from models trained on different modalities converge to a shared latent structure of the world. However, this hypothesis has largely been examined in vision and language, and it remains unclear whether time series participate in such convergence. We first examine this in a trimodal setting and find that independently pretrained time series, vision, and language encoders exhibit near-orthogonal geometry in the absence of explicit coupling. We then apply post-hoc alignment by training projection heads over frozen encoders using contrastive learning, and analyze the resulting representations with respect to geometry, scaling behavior, and dependence on information density and input modality characteristics. Our investigation reveals that overall alignment in contrastive representation spaces improves with model size, but this alignment is asymmetric: time series align more strongly with visual representations than with text, and images can act as effective intermediaries between time series and language. We further see that richer textual descriptions improve alignment only up to a threshold; training on denser captions does not lead to further improvement. Analogous effects are observed for visual representations. Our findings shed light on considerations for building multimodal systems involving non-conventional data modalities beyond vision and language.

Figures (24)

• Figure 1: Trimodal projections of a shared temporal process. A latent process $Z$ gives rise to a numeric time series, a visual line plot, and a textual description, each representing the same signal in values, geometry, and language. Modality-specific encoders $f_{\text{ts}}$, $f_{\text{img}}$, and $f_{\text{txt}}$ map inputs into representation spaces.
• Figure 2: Mean angular deviation between pretrained cross-modal representations on CaTS shows little inherent alignment.
• Figure 3: UMAP visualizations of representations after contrastive training on Flickr (top) and CaTS (bottom).
• Figure 4: Trimodal contrastive alignment framework. Frozen pretrained encoders independently map time series, visual plots, and text into their respective unimodal representation spaces. Trainable projection heads transform these representations into a shared embedding space. Alignment is learned via a symmetric contrastive objective applied jointly across all modality pairs (TS--IMG, TS--TXT, IMG--TXT).
• Figure 5: Scaling behavior of alignment across 34 trimodal configurations on CaTS as a function of total model size (in millions of parameters). Each subplot reports alignment quality for three modality pairs (TS--IMG, TS--TXT, IMG--TXT). Each point corresponds to a distinct encoder configuration. Dashed lines indicate linear trends with Pearson correlation coefficients reported in the legend.