Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Predicting Potential Customer Support Needs and Optimizing Search Ranking in a Two-Sided Marketplace

Do-kyum Kim, Han Zhao, Huiji Gao, Liwei He, Malay Haldar, Sanjeev Katariya

TL;DR

The paper tackles predicting CS support needs at booking time to reduce CS workload in a two sided marketplace. It formulates a binary classification using host, guest, and listing features, trains with a pairwise loss to optimize AUC, and calibrates probabilities with Platt scaling before injecting the CS risk signal into the Airbnb search ranking. Offline results show AUC above 0.73, and online A/B testing demonstrates a reduction in CS bookings by 3.7% and host cancellations by 2.7% without harming overall bookings, indicating more reliable guest-host matches. This approach offers a practical pathway to improve guest and host satisfaction while lowering operational costs through proactive CS risk management in search ranking.

Abstract

Airbnb is an online marketplace that connects hosts and guests to unique stays and experiences. When guests stay at homes booked on Airbnb, there are a small fraction of stays that lead to support needed from Airbnb's Customer Support (CS), which may cause inconvenience to guests and hosts and require Airbnb resources to resolve. In this work, we show that instances where CS support is needed may be predicted based on hosts and guests behavior. We build a model to predict the likelihood of CS support needs for each match of guest and host. The model score is incorporated into Airbnb's search ranking algorithm as one of the many factors. The change promotes more reliable matches in search results and significantly reduces bookings that require CS support.

Predicting Potential Customer Support Needs and Optimizing Search Ranking in a Two-Sided Marketplace

TL;DR

The paper tackles predicting CS support needs at booking time to reduce CS workload in a two sided marketplace. It formulates a binary classification using host, guest, and listing features, trains with a pairwise loss to optimize AUC, and calibrates probabilities with Platt scaling before injecting the CS risk signal into the Airbnb search ranking. Offline results show AUC above 0.73, and online A/B testing demonstrates a reduction in CS bookings by 3.7% and host cancellations by 2.7% without harming overall bookings, indicating more reliable guest-host matches. This approach offers a practical pathway to improve guest and host satisfaction while lowering operational costs through proactive CS risk management in search ranking.

Abstract

Airbnb is an online marketplace that connects hosts and guests to unique stays and experiences. When guests stay at homes booked on Airbnb, there are a small fraction of stays that lead to support needed from Airbnb's Customer Support (CS), which may cause inconvenience to guests and hosts and require Airbnb resources to resolve. In this work, we show that instances where CS support is needed may be predicted based on hosts and guests behavior. We build a model to predict the likelihood of CS support needs for each match of guest and host. The model score is incorporated into Airbnb's search ranking algorithm as one of the many factors. The change promotes more reliable matches in search results and significantly reduces bookings that require CS support.

Paper Structure

This paper contains 13 sections, 6 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Approach
  4. Binary Classification
  5. Maximizing Area under the ROC Curve
  6. Neural Networks
  7. Ranking Function
  8. Experiments
  9. Data Set
  10. Hyperparameter Search
  11. Offline Tuning of Ranking Function
  12. Online A/B Experiments
  13. Discussion

Figures (3)

  • Figure 1: Time from bookings to CS support needs
  • Figure 2: Calibration of model score using a Platt scaler. The line for calibrated score (red circle) is close to a diagonal line, which means the scaler provides well-calibrated probabilities.
  • Figure 3: Trade off between normalized DCGB and normalized DCGC as a multiplier $\alpha$ is varied. We need to increase both of the metrics. For the details of metrics, please refer to section \ref{['sec:tuning']}.