On Practical Diversified Recommendation with Controllable Category Diversity Framework

TL;DR

The paper tackles echo chamber effects in recommender systems by defining two diversity notions—U-diversity and N-diversity—and introducing CCDF, a two-stage framework with a tunable diversity parameter $K$ to explicitly control category diversity. Stage I uses the DeepU2C model for next-category prediction to select top-$K$ categories, while Stage II performs constrained item matching within those categories using a posterior weighted score that combines CTR, CVR, and CPR. The DeepU2C model combines a User Net, a Category Net, and a Wide Layer with a Multi-Head Self-Attention encoder, trained with a composite loss $L = L_c(u,c_t) + \lambda L_t(u,c_t,c_{nei})$, including a sampled softmax for $L_c$ and a triplet loss $L_t$ over neighboring categories. Offline results on Taobao and Alibaba.com show DeepU2C achieving superior HR@K on both U- and N-diversity tasks, while online A/B tests demonstrate that increasing $K$ yields higher exposure, clicks, and conversion, and enhances N-diversity exposure; these findings support a practical, production-ready approach to diversify recommendations and complement user search behavior.

Abstract

Recommender systems have made significant strides in various industries, primarily driven by extensive efforts to enhance recommendation accuracy. However, this pursuit of accuracy has inadvertently given rise to echo chamber/filter bubble effects. Especially in industry, it could impair user's experiences and prevent user from accessing a wider range of items. One of the solutions is to take diversity into account. However, most of existing works focus on user's explicit preferences, while rarely exploring user's non-interaction preferences. These neglected non-interaction preferences are especially important for broadening user's interests in alleviating echo chamber/filter bubble effects.Therefore, in this paper, we first define diversity as two distinct definitions, i.e., user-explicit diversity (U-diversity) and user-item non-interaction diversity (N-diversity) based on user historical behaviors. Then, we propose a succinct and effective method, named as Controllable Category Diversity Framework (CCDF) to achieve both high U-diversity and N-diversity simultaneously.Specifically, CCDF consists of two stages, User-Category Matching and Constrained Item Matching. The User-Category Matching utilizes the DeepU2C model and a combined loss to capture user's preferences in categories, and then selects the top-$K$ categories with a controllable parameter $K$.These top-$K$ categories will be used as trigger information in Constrained Item Matching. Offline experimental results show that our proposed DeepU2C outperforms state-of-the-art diversity-oriented methods, especially on N-diversity task. The whole framework is validated in a real-world production environment by conducting online A/B testing.

Figures (7)

• Figure 1: The proposed Controllable Category Diversity Framework (CCDF), which is divided into two stages. Top: User-Category Matching. Bottom: Constrained Item Matching
• Figure 2: Illustration of U-diversity and N-diversity. Different shapes refer to different categories. Top: user historical behavior. Middle: three top-5 recommendation cases. Bottom: U-diversity categories and N-diversity categories.
• Figure 3: The User-Category Matching. (a): definition of next category prediction task. (b): DeepU2C model architecture.
• Figure 4: The online implementation architecture of our proposed CCDF.
• Figure 5: Two real-world cases from online traffic. The upper part is user historical behaviors in chronological order, while the lower part is the recommended categories from CCDF. U-diversity categories are in blue and N-diversity categories are in red.