Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OCP: Orthogonal Constrained Projection for Sparse Scaling in Industrial Commodity Recommendation

Chen Sun, Beilin Xu, Boheng Tan, Jiacheng Wang, Yuefeng Sun, Rite Bo, Ying He, Yaqiang Zang, Pinghua Gong

Abstract

In industrial commodity recommendation systems, the representation quality of Item-Id vocabularies directly impacts the scalability and generalization ability of recommendation models. A key challenge is that traditional Item-Id vocabularies, when subjected to sparse scaling, suffer from low-frequency information interference, which restricts their expressive power for massive item sets and leads to representation collapse. To address this issue, we propose an Orthogonal Constrained Projection method to optimize embedding representation. By enforcing orthogonality, the projection constrains the backpropagation manifold, aligning the singular value spectrum of the learned embeddings with the orthogonal basis. This alignment ensures high singular entropy, thereby preserving isotropic generalized features while suppressing spurious correlations and overfitting to rare items. Empirical results demonstrate that OCP accelerates loss convergence and enhances the model's scalability; notably, it enables consistent performance gains when scaling up dense layers. Large-scale industrial deployment on JD.com further confirms its efficacy, yielding a 12.97% increase in UCXR and an 8.9% uplift in GMV, highlighting its robust utility for scaling up both sparse vocabularies and dense architectures.

OCP: Orthogonal Constrained Projection for Sparse Scaling in Industrial Commodity Recommendation

Abstract

In industrial commodity recommendation systems, the representation quality of Item-Id vocabularies directly impacts the scalability and generalization ability of recommendation models. A key challenge is that traditional Item-Id vocabularies, when subjected to sparse scaling, suffer from low-frequency information interference, which restricts their expressive power for massive item sets and leads to representation collapse. To address this issue, we propose an Orthogonal Constrained Projection method to optimize embedding representation. By enforcing orthogonality, the projection constrains the backpropagation manifold, aligning the singular value spectrum of the learned embeddings with the orthogonal basis. This alignment ensures high singular entropy, thereby preserving isotropic generalized features while suppressing spurious correlations and overfitting to rare items. Empirical results demonstrate that OCP accelerates loss convergence and enhances the model's scalability; notably, it enables consistent performance gains when scaling up dense layers. Large-scale industrial deployment on JD.com further confirms its efficacy, yielding a 12.97% increase in UCXR and an 8.9% uplift in GMV, highlighting its robust utility for scaling up both sparse vocabularies and dense architectures.
Paper Structure (23 sections, 6 equations, 4 figures, 3 tables)

This paper contains 23 sections, 6 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preliminary
  4. Embedding Collapse in Scaled Models
  5. Orthogonal Constrained Projection Layer
  6. Orthogonal Constraint on Stiefel Manifold
  7. Analysis of Singular Value Preservation
  8. Experiments
  9. Experimental Setup
  10. Baseline
  11. Evaluation Metric
  12. Training and Evaluation Protocol
  13. Reproducibility Notes
  14. Overall Performance
  15. Singular Entropy Study
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Overview of OCP.
  • Figure 2: Comparison of singular value distributions. Orders 1--10 and 55--64 represent the top-10 and bottom-10 singular values of the $d=64$ embedding matrix, respectively.
  • Figure 3: Dense scaling law on OxygenREC with different model sizes. As parameters increase from 0.7B to 1.7B and 3.2B, OCP achieves lower final loss.
  • Figure 4: Sparse scaling law on OxygenREC with different vocabulary sizes. As vocabulary increases from 1e ($10^8$) to 2e, 5e, and 10e, OCP achieves a lower final loss.