HMDN: Hierarchical Multi-Distribution Network for Click-Through Rate Prediction

TL;DR

HMDN introduces a flexible framework to model hierarchical relationships across mixed multi-distributions in CTR prediction by refining distribution representations through multi-level residual quantization (HMDRR) and integrating the resulting hierarchical vector $s_D$ into existing backbones like Mixture-of-Experts and Dynamic-Weight models. The HMDRR module yields coarse-to-fine hierarchical representations via residual quantization, which improves gating and weighting in MoE, and guides input-level modulation in DW models. Empirical results on the Ali-CCP and an industrial dataset demonstrate consistent improvements over strong single-distribution baselines, with ablations showing explicit vs implicit extraction and optimal codebook depth around $D=6$. The work highlights the importance of hierarchical multi-distribution modeling for scalable, efficient CTR prediction and offers a practical, model-agnostic refinement step that can be plugged into existing systems.

Abstract

As the recommendation service needs to address increasingly diverse distributions, such as multi-population, multi-scenario, multitarget, and multi-interest, more and more recent works have focused on multi-distribution modeling and achieved great progress. However, most of them only consider modeling in a single multi-distribution manner, ignoring that mixed multi-distributions often coexist and form hierarchical relationships. To address these challenges, we propose a flexible modeling paradigm, named Hierarchical Multi-Distribution Network (HMDN), which efficiently models these hierarchical relationships and can seamlessly integrate with existing multi-distribution methods, such as Mixture of-Experts (MoE) and Dynamic-Weight (DW) models. Specifically, we first design a hierarchical multi-distribution representation refinement module, employing a multi-level residual quantization to obtain fine-grained hierarchical representation. Then, the refined hierarchical representation is integrated into the existing single multi-distribution models, seamlessly expanding them into mixed multi-distribution models. Experimental results on both public and industrial datasets validate the effectiveness and flexibility of HMDN.

Figures (2)

• Figure 1: Illustration of the proposed HMDM framework. ($\otimes$) denotes the element-wise multiplication operation.
• Figure 2: Effect of different codebook depth D.