Towards Automated Model Design on Recommender Systems

TL;DR

This work introduces a novel paradigm that utilizes weight sharing to explore abundant solution spaces and creates a large supernet to search for optimal architectures and co-design strategies to address the challenges of data multimodality and heterogeneity in the recommendation domain.

Abstract

The increasing popularity of deep learning models has created new opportunities for developing AI-based recommender systems. Designing recommender systems using deep neural networks requires careful architecture design, and further optimization demands extensive co-design efforts on jointly optimizing model architecture and hardware. Design automation, such as Automated Machine Learning (AutoML), is necessary to fully exploit the potential of recommender model design, including model choices and model-hardware co-design strategies. We introduce a novel paradigm that utilizes weight sharing to explore abundant solution spaces. Our paradigm creates a large supernet to search for optimal architectures and co-design strategies to address the challenges of data multi-modality and heterogeneity in the recommendation domain. From a model perspective, the supernet includes a variety of operators, dense connectivity, and dimension search options. From a co-design perspective, it encompasses versatile Processing-In-Memory (PIM) configurations to produce hardware-efficient models. Our solution space's scale, heterogeneity, and complexity pose several challenges, which we address by proposing various techniques for training and evaluating the supernet. Our crafted models show promising results on three Click-Through Rates (CTR) prediction benchmarks, outperforming both manually designed and AutoML-crafted models with state-of-the-art performance when focusing solely on architecture search. From a co-design perspective, we achieve 2x FLOPs efficiency, 1.8x energy efficiency, and 1.5x performance improvements in recommender models.

Figures (8)

• Figure 1: NASRec search space enables a full architecture search on dense connectivity of blocks, dense/sparse operators, and mergers that fuse dense/sparse representations.
• Figure 2: Operation principle of ReRAM-based in-memory computing.
• Figure 3: Single-operator Any-connection path sampling combines the advantages of the first two sampling strategies.
• Figure 4: Operator-balancing interaction ensures linear parameter consumption and balance building operators.
• Figure 5: Best NASRec models discovered on Avazu.