DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction

TL;DR

The paper addresses CTR prediction by recognizing that heterogeneous interaction modules provide complementary information across datasets. It introduces DHEN, a deep hierarchical ensemble that stacks layers of diverse interaction modules to learn a hierarchy of higher-order feature interactions and their correlations. A co-designed training system, Hybrid Sharded Data Parallel (HSDP) built on ZionEX, enables efficient training of these large models via embedding table sharding and selective data-parallelism. On a large industrial CTR dataset, DHEN delivers up to a 0.27 percentage-point improvement in Normalized Entropy ($NE$) and up to 1.2x training throughput over state-of-the-art baselines, illustrating both predictive gains and practical scalability.

Abstract

Learning feature interactions is important to the model performance of online advertising services. As a result, extensive efforts have been devoted to designing effective architectures to learn feature interactions. However, we observe that the practical performance of those designs can vary from dataset to dataset, even when the order of interactions claimed to be captured is the same. That indicates different designs may have different advantages and the interactions captured by them have non-overlapping information. Motivated by this observation, we propose DHEN - a deep and hierarchical ensemble architecture that can leverage strengths of heterogeneous interaction modules and learn a hierarchy of the interactions under different orders. To overcome the challenge brought by DHEN's deeper and multi-layer structure in training, we propose a novel co-designed training system that can further improve the training efficiency of DHEN. Experiments of DHEN on large-scale dataset from CTR prediction tasks attained 0.27\% improvement on the Normalized Entropy (NE) of prediction and 1.2x better training throughput than state-of-the-art baseline, demonstrating their effectiveness in practice.

Figures (7)

• Figure 1: An general hierarchical ensemble building block in DHEN.
• Figure 2: A Two-layer two-module hierarchical ensemble (left) and its expanded details (right). A general DHEN can be expressed as a mixture of multiple high-order interactions. We omit the potential dense feature input for the interaction modules in this figure for clarity.
• Figure 3: The feature processing layer in DHEN
• Figure 3: DHEN achieves better scaling efficiency than MLP scaling by MoE.
• Figure 4: Training strategy for DHEN (4 GPUs shown).