Breaking the Barrier: Utilizing Large Language Models for Industrial Recommendation Systems through an Inferential Knowledge Graph

TL;DR

The paper tackles industrial recommendation challenges such as cold-start items and evolving item catalogs by proposing LLM-KERec, a hybrid system that builds a complementary knowledge graph via an LLM and an entity-extraction workflow. It introduces an Entity Dict and a BERT-CRF-based entity extractor, constructs a cost-aware Entity-Entity graph through LLM reasoning, and trains an E-E-I weight decision model using real exposure data. The ranking stage uses a dual-view graph neural approach with contrastive learning to fuse first-order substitutable and second-order complementary signals, followed by an integration stage that augments the recall and fine-ranking with complementary items. Experiments on three Alipay industrial datasets show consistent offline AUC gains and online improvements in conversions and GMV, validating the method’s practicality and impact in dynamic e-commerce environments.

Abstract

Recommendation systems are widely used in e-commerce websites and online platforms to address information overload. However, existing systems primarily rely on historical data and user feedback, making it difficult to capture user intent transitions. Recently, Knowledge Base (KB)-based models are proposed to incorporate expert knowledge, but it struggle to adapt to new items and the evolving e-commerce environment. To address these challenges, we propose a novel Large Language Model based Complementary Knowledge Enhanced Recommendation System (LLM-KERec). It introduces an entity extractor that extracts unified concept terms from item and user information. To provide cost-effective and reliable prior knowledge, entity pairs are generated based on entity popularity and specific strategies. The large language model determines complementary relationships in each entity pair, constructing a complementary knowledge graph. Furthermore, a new complementary recall module and an Entity-Entity-Item (E-E-I) weight decision model refine the scoring of the ranking model using real complementary exposure-click samples. Extensive experiments conducted on three industry datasets demonstrate the significant performance improvement of our model compared to existing approaches. Additionally, detailed analysis shows that LLM-KERec enhances users' enthusiasm for consumption by recommending complementary items. In summary, LLM-KERec addresses the limitations of traditional recommendation systems by incorporating complementary knowledge and utilizing a large language model to capture user intent transitions, adapt to new items, and enhance recommendation efficiency in the evolving e-commerce landscape.

Figures (5)

• Figure 1: Overall framework of our proposed LLM-KERec System.
• Figure 2: Extracting entites from item information and user bills.
• Figure 3: Long-tail distribution in Entity Dict.
• Figure 4: Overall framework of Two-stage Complementary Knowledge Enhancement Procedure.
• Figure 5: The relative improvement of conversion rate (CVR) for randomly sampled complementary pairs in LLM-KERec compared to the baseline.