CELA: Cost-Efficient Language Model Alignment for CTR Prediction

TL;DR

CELA incorporates textual features and language models while preserving the collaborative filtering capabilities of ID-based models, and can be equipped with plug-and-play textual features, with item-level alignment enhancing the utilization of external information while maintaining training and inference efficiency.

Abstract

Click-Through Rate (CTR) prediction holds a paramount position in recommender systems. The prevailing ID-based paradigm underperforms in cold-start scenarios due to the skewed distribution of feature frequency. Additionally, the utilization of a single modality fails to exploit the knowledge contained within textual features. Recent efforts have sought to mitigate these challenges by integrating Pre-trained Language Models (PLMs). They design hard prompts to structure raw features into text for each interaction and then apply PLMs for text processing. With external knowledge and reasoning capabilities, PLMs extract valuable information even in cases of sparse interactions. Nevertheless, compared to ID-based models, pure text modeling degrades the efficacy of collaborative filtering, as well as feature scalability and efficiency during both training and inference. To address these issues, we propose \textbf{C}ost-\textbf{E}fficient \textbf{L}anguage Model \textbf{A}lignment (\textbf{CELA}) for CTR prediction. CELA incorporates textual features and language models while preserving the collaborative filtering capabilities of ID-based models. This model-agnostic framework can be equipped with plug-and-play textual features, with item-level alignment enhancing the utilization of external information while maintaining training and inference efficiency. Through extensive offline experiments, CELA demonstrates superior performance compared to state-of-the-art methods. Furthermore, an online A/B test conducted on an industrial App recommender system showcases its practical effectiveness, solidifying the potential for real-world applications of CELA.

Figures (7)

• Figure 1: ID-Based Models vs. Pre-trained Language Models (PLMs) for CTR prediction. The former converts tabular data into one-hot vectors, capturing collaborative signals without external knowledge. The latter populates data into a hard prompt, and then feeds truncated texts into the PLM, leveraging external world knowledge for data interpretation.
• Figure 2: The overall framework of CELA. The first stage pre-trains a PLM on domain-specific item texts. In the second stage, an ID-based model is developed and item text representations from the PLM are aligned with the item-side feature embeddings of the ID-based model in latent space. The third stage merges aligned text representations with non-textual features for training a new ID-based model. The final two stages are executed alternately, as denoted by the green dotted line.
• Figure 3: Performance comparison w.r.t. long-tail items on Amazon. The bar graph represents the number of interactions within test data for each group, while the line chart represents the AUC.
• Figure 4: Performance comparison of CELAs with different PLMs on Amazon. The backbone is indicated by N/A.
• Figure 5: Performance with alignment w.r.t. popular items, evaluated on Amazon. The $x$-axis represents the top percentile of popular items.