Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender

Zhaoqi Zhang, Haolei Pei, Jun Guo, Tianyu Wang, Yufei Feng, Hui Sun, Shaowei Liu, Aixin Sun

TL;DR

OneTrans proposes a unified Transformer backbone that jointly handles user-behavior sequence modeling and feature interaction for industrial recommender systems. By unifying sequential and non-sequential features into a single token sequence and employing a pyramid stack with mixed parameterization, it enables bidirectional information exchange and efficient scaling. Cross-request KV caching and LLMS-style optimizations (FlashAttention, mixed precision) deliver substantial runtime and memory savings, while experiments show consistent performance gains and significant online Lift in GMV and orders with maintained latency. The approach establishes a practical scaling paradigm for RecSys by reusing mature LLM engineering techniques within a unified, production-friendly architecture.

Abstract

In recommendation systems, scaling up feature-interaction modules (e.g., Wukong, RankMixer) or user-behavior sequence modules (e.g., LONGER) has achieved notable success. However, these efforts typically proceed on separate tracks, which not only hinders bidirectional information exchange but also prevents unified optimization and scaling. In this paper, we propose OneTrans, a unified Transformer backbone that simultaneously performs user-behavior sequence modeling and feature interaction. OneTrans employs a unified tokenizer to convert both sequential and non-sequential attributes into a single token sequence. The stacked OneTrans blocks share parameters across similar sequential tokens while assigning token-specific parameters to non-sequential tokens. Through causal attention and cross-request KV caching, OneTrans enables precomputation and caching of intermediate representations, significantly reducing computational costs during both training and inference. Experimental results on industrial-scale datasets demonstrate that OneTrans scales efficiently with increasing parameters, consistently outperforms strong baselines, and yields a 5.68% lift in per-user GMV in online A/B tests.

OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender

TL;DR

OneTrans proposes a unified Transformer backbone that jointly handles user-behavior sequence modeling and feature interaction for industrial recommender systems. By unifying sequential and non-sequential features into a single token sequence and employing a pyramid stack with mixed parameterization, it enables bidirectional information exchange and efficient scaling. Cross-request KV caching and LLMS-style optimizations (FlashAttention, mixed precision) deliver substantial runtime and memory savings, while experiments show consistent performance gains and significant online Lift in GMV and orders with maintained latency. The approach establishes a practical scaling paradigm for RecSys by reusing mature LLM engineering techniques within a unified, production-friendly architecture.

Abstract

In recommendation systems, scaling up feature-interaction modules (e.g., Wukong, RankMixer) or user-behavior sequence modules (e.g., LONGER) has achieved notable success. However, these efforts typically proceed on separate tracks, which not only hinders bidirectional information exchange but also prevents unified optimization and scaling. In this paper, we propose OneTrans, a unified Transformer backbone that simultaneously performs user-behavior sequence modeling and feature interaction. OneTrans employs a unified tokenizer to convert both sequential and non-sequential attributes into a single token sequence. The stacked OneTrans blocks share parameters across similar sequential tokens while assigning token-specific parameters to non-sequential tokens. Through causal attention and cross-request KV caching, OneTrans enables precomputation and caching of intermediate representations, significantly reducing computational costs during both training and inference. Experimental results on industrial-scale datasets demonstrate that OneTrans scales efficiently with increasing parameters, consistently outperforms strong baselines, and yields a 5.68% lift in per-user GMV in online A/B tests.

Paper Structure

This paper contains 26 sections, 13 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. OneTrans Framework Overview
  5. Features and Tokenization
  6. Non-Sequential Tokenization
  7. Sequential Tokenization
  8. OneTrans Block
  9. Mixed (shared/token-specific) Causal Attention
  10. Mixed (shared/token-specific) FFN
  11. Pyramid Stack
  12. Training and Deployment Optimization
  13. Cross Request KV Caching
  14. Unified LLM Optimizations.
  15. Experiments
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Architectural comparison. (a) Conventional encode-then-interaction pipeline encodes sequential features and merges non-sequential features before a post-hoc feature interaction block. (b)OneTrans performs joint modeling of both sequential and non-sequential features within a single OneTrans (Transformer-style) stack.
  • Figure 2: System Architecture.(a)OneTrans overview. Sequential (S, blue) and non-sequential (NS, orange) features are tokenized separately. After inserting [SEP] between user behavior sequences, the unified token sequence is fed into stacked OneTrans Pyramid Blocks that progressively shrink the token length until it matches the number of NS tokens. (b)OneTrans Block: a causal pre-norm Transformer Block with RMSNorm, Mixed Causal Attention and Mixed FFN. (c) "Mixed" = mixed parameterization: S tokens share one set of QKV/FFN weights, while each NS token receives its own token-specific QKV/FFN.
  • Figure 3: Comparison of trade-off and scaling law.