AIF: Asynchronous Inference Framework for Cost-Effective Pre-Ranking

TL;DR

The paper tackles latency and redundancy in industrial pre-ranking by introducing the Asynchronous Inference Framework (AIF), which decouples interaction-independent components from real-time prediction and applies online asynchronous user-side and nearline item-side inference. By combining BEA (Bridge Embedding Approximation) for expanded interactions and LSH-based long-term user behavior modeling, AIF significantly increases model capacity without incurring prohibitive latency, achieving an online CTR improvement of +8.72% and RPM of +5.80% in Taobao display advertising. The authors demonstrate both model-level gains (GAUC +7.29pt, HR@100 gains) and system-level efficiency (latency and QPS improvements) across extensive offline and online experiments, with deployment since October 2023. The work presents a practical, co-designed framework and model adaptations that translate to meaningful performance and efficiency benefits in large-scale recommender systems.

Abstract

In industrial recommendation systems, pre-ranking models based on deep neural networks (DNNs) commonly adopt a sequential execution framework: feature fetching and model forward computation are triggered only after receiving candidates from the upstream retrieval stage. This design introduces inherent bottlenecks, including redundant computations of identical users/items and increased latency due to strictly sequential operations, which jointly constrain the model's capacity and system efficiency. To address these limitations, we propose the Asynchronous Inference Framework (AIF), a cost-effective computational architecture that decouples interaction-independent components, those operating within a single user or item, from real-time prediction. AIF reorganizes the model inference process by performing user-side computations in parallel with the retrieval stage and conducting item-side computations in a nearline manner. This means that interaction-independent components are calculated just once and completed before the real-time prediction phase of the pre-ranking stage. As a result, AIF enhances computational efficiency and reduces latency, freeing up resources to significantly improve the feature set and model architecture of interaction-independent components. Moreover, we delve into model design within the AIF framework, employing approximated methods for interaction-dependent components in online real-time predictions. By co-designing both the framework and the model, our solution achieves notable performance gains without significantly increasing computational and latency costs. This has enabled the successful deployment of AIF in the Taobao display advertising system.

Figures (6)

• Figure 1: An illustration of the typical multi-stage cascade architecture for industrial recommender systems.
• Figure 2: A comparison between sequential inference framework and AIF. In AIF, the intra-computation of user-side features and the user-side component of cross features(the blue part) are performed through online asynchronous inference, while the intra-computation of item-side features(the green part) utilizes nearline asynchronous inference.
• Figure 3: Asynchronous online inference for user-side computations.
• Figure 4: Asynchronous nearline inference for item-side computations.
• Figure 5: Illustration of pre-caching mechanism for SIM features.