An Interpretable Ensemble of Graph and Language Models for Improving Search Relevance in E-Commerce

TL;DR

This paper tackles the practical challenge of improving search relevance in multilingual, real-world e-commerce by proposing PP-GLAM, a plug-and-play, interpretable ensemble of language models and relational GNNs. It uses a gradient-boosted decision tree (GBDT) with SHAP-based additive explanations to autonomously select and combine semantic and behavioral signals, while handling constraints on computation and latency. Empirical results on a large, multi-regional ESCI dataset show PP-GLAM outperforms state-of-the-art baselines and a proprietary model, with strong interpretability and a deployment strategy tailored to dynamic data. The approach offers a practical pathway for industry adoption by balancing accuracy, efficiency, and explainability in a heterogeneous, real-world setting.

Abstract

The problem of search relevance in the E-commerce domain is a challenging one since it involves understanding the intent of a user's short nuanced query and matching it with the appropriate products in the catalog. This problem has traditionally been addressed using language models (LMs) and graph neural networks (GNNs) to capture semantic and inter-product behavior signals, respectively. However, the rapid development of new architectures has created a gap between research and the practical adoption of these techniques. Evaluating the generalizability of these models for deployment requires extensive experimentation on complex, real-world datasets, which can be non-trivial and expensive. Furthermore, such models often operate on latent space representations that are incomprehensible to humans, making it difficult to evaluate and compare the effectiveness of different models. This lack of interpretability hinders the development and adoption of new techniques in the field. To bridge this gap, we propose Plug and Play Graph LAnguage Model (PP-GLAM), an explainable ensemble of plug and play models. Our approach uses a modular framework with uniform data processing pipelines. It employs additive explanation metrics to independently decide whether to include (i) language model candidates, (ii) GNN model candidates, and (iii) inter-product behavioral signals. For the task of search relevance, we show that PP-GLAM outperforms several state-of-the-art baselines as well as a proprietary model on real-world multilingual, multi-regional e-commerce datasets. To promote better model comprehensibility and adoption, we also provide an analysis of the explainability and computational complexity of our model. We also provide the public codebase and provide a deployment strategy for practical implementation.

Figures (4)

• Figure 1: An illustration of the search relevance problem. Using the semantic information of queries and products along with the behavioral relationships between them, the goal is to classify the degree of relevance for unseen query-product pairs (labeled with question marks) as exact, substitute, complement, or irrelevant.
• Figure 2: An overview of the PP-GLAM architecture. The model contains four modules: (i) Data Processing handles the pre-computation of graph neighborhoods and the de-noising of the product information, (ii) Model Training uses the training samples with query-product pairs and corresponding ground-truth labels to learn the parameters of the GBDT-based ensemble containing different language models and graph neural networks (GNNs), (iii) Model Selection module utilizes interpretable SHAP values to eliminate low-performing models based on the constraints of the inference setup, and (iv) Model Inference loads the selected set of models into memory for inference on new batches.
• Figure 3: Correlation between edges of the dataset. Note that dense signals such as impressions and clicks have a low correlation with target edges of exact, substitute, complement, and irrelevant, whereas sparser signals such as adds and purchases are highly correlated. Hence, we must carefully consider the reliability-availability trade-off in practice.
• Figure 4: SHAP value-based additive contribution of different features to the overall performance of our model. (a) provides the contribution of different models, and (b) shows the contribution of different relations.