Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Reciprocal Recommender Systems: Metrics, Formulation, and Method

Chen Yang, Sunhao Dai, Yupeng Hou, Wayne Xin Zhao, Jun Xu, Yang Song, Hengshu Zhu

TL;DR

This work reframes reciprocal recommender systems through a holistic evaluation lens and a causal modeling paradigm. It introduces five evaluation metrics (CRecall, CPrecision, SRecall, SPrecision, RNDCG) to capture overall coverage, bilateral stability, and balanced ranking, addressing limitations of per-side metrics. The authors formulate RRS as bilateral interventions within the potential outcome framework and propose CRRS, a model-agnostic method with a reranking strategy for vacant slots, optimized via a two-stage pre-training and counterfactual learning pipeline. Experiments on recruitment and dating datasets demonstrate improved holistic performance and meaningful insights into redundancy, underlining the practical value of holistic metrics and causal modeling for RRS. The work provides accessible code and datasets, offering a concrete path to more efficient and fair bilateral matching in real-world two-sided markets.

Abstract

Reciprocal recommender systems~(RRS), conducting bilateral recommendations between two involved parties, have gained increasing attention for enhancing matching efficiency. However, the majority of existing methods in the literature still reuse conventional ranking metrics to separately assess the performance on each side of the recommendation process. These methods overlook the fact that the ranking outcomes of both sides collectively influence the effectiveness of the RRS, neglecting the necessity of a more holistic evaluation and a capable systemic solution. In this paper, we systemically revisit the task of reciprocal recommendation, by introducing the new metrics, formulation, and method. Firstly, we propose five new evaluation metrics that comprehensively and accurately assess the performance of RRS from three distinct perspectives: overall coverage, bilateral stability, and balanced ranking. These metrics provide a more holistic understanding of the system's effectiveness and enable a comprehensive evaluation. Furthermore, we formulate the RRS from a causal perspective, formulating recommendations as bilateral interventions, which can better model the decoupled effects of potential influencing factors. By utilizing the potential outcome framework, we further develop a model-agnostic causal reciprocal recommendation method that considers the causal effects of recommendations. Additionally, we introduce a reranking strategy to maximize matching outcomes, as measured by the proposed metrics. Extensive experiments on two real-world datasets from recruitment and dating scenarios demonstrate the effectiveness of our proposed metrics and approach. The code and dataset are available at: https://github.com/RUCAIBox/CRRS.

Revisiting Reciprocal Recommender Systems: Metrics, Formulation, and Method

TL;DR

This work reframes reciprocal recommender systems through a holistic evaluation lens and a causal modeling paradigm. It introduces five evaluation metrics (CRecall, CPrecision, SRecall, SPrecision, RNDCG) to capture overall coverage, bilateral stability, and balanced ranking, addressing limitations of per-side metrics. The authors formulate RRS as bilateral interventions within the potential outcome framework and propose CRRS, a model-agnostic method with a reranking strategy for vacant slots, optimized via a two-stage pre-training and counterfactual learning pipeline. Experiments on recruitment and dating datasets demonstrate improved holistic performance and meaningful insights into redundancy, underlining the practical value of holistic metrics and causal modeling for RRS. The work provides accessible code and datasets, offering a concrete path to more efficient and fair bilateral matching in real-world two-sided markets.

Abstract

Reciprocal recommender systems~(RRS), conducting bilateral recommendations between two involved parties, have gained increasing attention for enhancing matching efficiency. However, the majority of existing methods in the literature still reuse conventional ranking metrics to separately assess the performance on each side of the recommendation process. These methods overlook the fact that the ranking outcomes of both sides collectively influence the effectiveness of the RRS, neglecting the necessity of a more holistic evaluation and a capable systemic solution. In this paper, we systemically revisit the task of reciprocal recommendation, by introducing the new metrics, formulation, and method. Firstly, we propose five new evaluation metrics that comprehensively and accurately assess the performance of RRS from three distinct perspectives: overall coverage, bilateral stability, and balanced ranking. These metrics provide a more holistic understanding of the system's effectiveness and enable a comprehensive evaluation. Furthermore, we formulate the RRS from a causal perspective, formulating recommendations as bilateral interventions, which can better model the decoupled effects of potential influencing factors. By utilizing the potential outcome framework, we further develop a model-agnostic causal reciprocal recommendation method that considers the causal effects of recommendations. Additionally, we introduce a reranking strategy to maximize matching outcomes, as measured by the proposed metrics. Extensive experiments on two real-world datasets from recruitment and dating scenarios demonstrate the effectiveness of our proposed metrics and approach. The code and dataset are available at: https://github.com/RUCAIBox/CRRS.
Paper Structure (24 sections, 10 equations, 6 figures, 4 tables)

This paper contains 24 sections, 10 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Definition
  4. Revisiting the Evaluation Metrics
  5. Proposed Metrics
  6. Effectiveness of the Proposed Metrics
  7. Proposed Method
  8. Recommendations as Bilateral Treatments
  9. Causal Reciprocal Recommender System
  10. Potential Outcome Framework
  11. Causal Effect Modeling and Ranking
  12. Reranking Strategy with Vacant Slots
  13. Optimization
  14. First Stage: Pre-training.
  15. Second Stage: Counterfactual Learning.
  16. ...and 9 more sections

Figures (6)

  • Figure 1: Causal graph for (a) recommendation as treatment in convention RS; (b) reciprocal recommendation as bilateral treatments in RRS. U: user, I: item, R: recommendation, $\mathbf{R_A}$ and $\mathbf{R_B}$: recommendations made for each side, Y: ranking score (e.g., the probability of matching). For simplicity, we ignore all confounding factors.
  • Figure 2: Three distinct recommendation cases with identical matching relationships. In this scenario, we make a top-1 recommendation for four users ($a_1$, $a_2$, $b_1$, and $b_2$) and then report the average Recall, CRecall, SRecall, and the count of successful matching pairs.
  • Figure 3: An illustration of the proposed framework CRRS. We represent three treatment assignments with red, purple, and blue. Each assignment signifies a distinct dataset and outcome obtained under the treatment conditions.
  • Figure 4: The learning algorithm of CRRS.
  • Figure 5: Comparative analysis between traditional and proposed metrics. We employ the same model BPRMF on Dating and adjust the redundant recommendations by manually converting them into non-redundant ones and vice versa. In three cases, the models demonstrate identical traditional metrics yet display completely divergent overall performance.
  • ...and 1 more figures

Theorems & Definitions (2)

  • definition 1: Bilateral Treatments
  • definition 2: Potential Outcome for RRS