Time2Vec: Learning a Vector Representation of Time

TL;DR

This work tackles how to represent time within predictive models by introducing Time2Vec, a learnable vector embedding that combines a linear time term with multiple sine components. The representation is model-agnostic and designed to capture both periodic and non-periodic temporal patterns while being invariant to time rescaling. Across diverse datasets and architectures, Time2Vec improves performance over traditional time features, with learned frequencies providing interpretable and effective temporal encodings. The findings suggest Time2Vec as a practical tool for enhancing sequence and event-based models, with potential for broader theoretical and optimization-focused future work.

Abstract

Time is an important feature in many applications involving events that occur synchronously and/or asynchronously. To effectively consume time information, recent studies have focused on designing new architectures. In this paper, we take an orthogonal but complementary approach by providing a model-agnostic vector representation for time, called Time2Vec, that can be easily imported into many existing and future architectures and improve their performances. We show on a range of models and problems that replacing the notion of time with its Time2Vec representation improves the performance of the final model.

Figures (14)

• Figure 1: Comparing LSTM+T and LSTM+Time2Vec on several datasets.
• Figure 2: Comparing TLSTM1 and TLSTM3 on Last.FM and CiteULike in terms of Recall@10 with and without Time2Vec.
• Figure 3: The models learned for our synthesized dataset explained in Subsection \ref{['syn-exp-section']} before the final activation. The red dots represent the points to be classified as $1$.
• Figure 4: (a) Initial vs. (b) learned weights and frequencies for our synthesized dataset.
• Figure 5: An ablation study of several components in Time2Vec. (a) Comparing different activation functions for Time2Vec on Event-MNIST. Sigmoid and Tanh almost overlap. (b) Comparing frequencies fixed to equally-spaced values, frequencies fixed according to positional encoding vaswani2017attention, and learned frequencies on Event-MNIST. (c) A histogram of the frequencies learned in Time2Vec for Event-MNIST. The x-axis represents frequency intervals and the y-axis represents the number of frequencies in that interval. (d) The performance of TLSTM3+Time2Vec on CiteULike in terms of Recall@10 with and without the linear term.

Theorems & Definitions (3)

• Proposition 1
• Proposition 1
• proof