Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rotate Both Ways: Time-and-Order RoPE for Generative Recommendation

Xiaokai Wei, Jiajun Wu, Daiyao Yi, Reza Shirkavand, Michelle Gong

TL;DR

The paper tackles the challenge of encoding both the discrete order and wall-clock time of user interactions in generative recommender systems. It introduces Time-and-Order RoPE (TO-RoPE), a unified angular formulation that blends index and time as plane-wise rotations, with instantiations including early fusion, split-by-dimension, and split-by-head. Empirical results on MovieLens-20M and a large proprietary dataset show that TO-RoPE variants consistently outperform absolute embeddings and relative-bias baselines, with split-by-dim and split-by-head achieving the best performance and offering practical capacity-allocation guidance. The work demonstrates that rotary embeddings can serve as a simple, scalable, deployment-friendly foundation for modeling burstiness, rhythms, and calendar effects in real-world recommendations, enabling better long-range context and temporally aware personalization.

Abstract

Generative recommenders, typically transformer-based autoregressive models, predict the next item or action from a user's interaction history. Their effectiveness depends on how the model represents where an interaction event occurs in the sequence (discrete index) and when it occurred in wall-clock time. Prevailing approaches inject time via learned embeddings or relative attention biases. In this paper, we argue that RoPE-based approaches, if designed properly, can be a stronger alternative for jointly modeling temporal and sequential information in user behavior sequences. While vanilla RoPE in LLMs considers only token order, generative recommendation requires incorporating both event time and token index. To address this, we propose Time-and-Order RoPE (TO-RoPE), a family of rotary position embedding designs that treat index and time as angle sources shaping the query-key geometry directly. We present three instantiations: early fusion, split-by-dim, and split-by-head. Extensive experiments on both publicly available datasets and a proprietary industrial dataset show that TO-RoPE variants consistently improve accuracy over existing methods for encoding time and index. These results position rotary embeddings as a simple, principled, and deployment-friendly foundation for generative recommendation.

Rotate Both Ways: Time-and-Order RoPE for Generative Recommendation

TL;DR

The paper tackles the challenge of encoding both the discrete order and wall-clock time of user interactions in generative recommender systems. It introduces Time-and-Order RoPE (TO-RoPE), a unified angular formulation that blends index and time as plane-wise rotations, with instantiations including early fusion, split-by-dimension, and split-by-head. Empirical results on MovieLens-20M and a large proprietary dataset show that TO-RoPE variants consistently outperform absolute embeddings and relative-bias baselines, with split-by-dim and split-by-head achieving the best performance and offering practical capacity-allocation guidance. The work demonstrates that rotary embeddings can serve as a simple, scalable, deployment-friendly foundation for modeling burstiness, rhythms, and calendar effects in real-world recommendations, enabling better long-range context and temporally aware personalization.

Abstract

Generative recommenders, typically transformer-based autoregressive models, predict the next item or action from a user's interaction history. Their effectiveness depends on how the model represents where an interaction event occurs in the sequence (discrete index) and when it occurred in wall-clock time. Prevailing approaches inject time via learned embeddings or relative attention biases. In this paper, we argue that RoPE-based approaches, if designed properly, can be a stronger alternative for jointly modeling temporal and sequential information in user behavior sequences. While vanilla RoPE in LLMs considers only token order, generative recommendation requires incorporating both event time and token index. To address this, we propose Time-and-Order RoPE (TO-RoPE), a family of rotary position embedding designs that treat index and time as angle sources shaping the query-key geometry directly. We present three instantiations: early fusion, split-by-dim, and split-by-head. Extensive experiments on both publicly available datasets and a proprietary industrial dataset show that TO-RoPE variants consistently improve accuracy over existing methods for encoding time and index. These results position rotary embeddings as a simple, principled, and deployment-friendly foundation for generative recommendation.
Paper Structure (30 sections, 25 equations, 4 figures, 5 tables)

This paper contains 30 sections, 25 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Incorporate Time in User History Modeling
  4. Position Embedding for LLMs
  5. Preliminaries
  6. Rotary Position Embeddings (RoPE)
  7. Rotary Embeddings: From Index to Time
  8. Why RoPE Fits Generative Recommendation
  9. Limitations of Single Source RoPE
  10. Staleness vs. adjacency (close in index but stale).
  11. Temporal periodicity (calendar effects).
  12. Methodology
  13. Time-and-Order RoPE: A Unified View
  14. Formulation
  15. Special cases (instantiations).
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Illustration of typical user behavior sequence. Several time patterns conoccur, including short-term burstness, long-term patterns and periodicity.
  • Figure 2: First row: single source RoPE. Second row: TO-RoPE variants which can incorporate both index and time information
  • Figure 3: Attention distances for different positonal encoding variants. We measure the average attention distance by weighting query-key distance with attention probabilities.
  • Figure 4: Attention entropy for different positonal encoding variants. Higher entropy indicates more tokens are involved in attention.