Analyzing Customer-Facing Vendor Experiences with Time Series Forecasting and Monte Carlo Techniques

TL;DR

The paper tackles the problem of deciding if and when to disable a problematic external vendor to preserve customer experience during outages. It proposes a data-driven pipeline that integrates a Baseline baseline forecast with multiplicative seasonality and a Trend via MAP, a Wired-On forecast that uses a Double Exponential Smoothing availability model plus a Monte Carlo simulation of retry/switch decisions, and a Wired-Off forecast based on a simple linear relation between the disabled vendor's baseline and the residual volume across other vendors. Validation on historical incidents shows the approach yields reliable forecasts and enables earlier, data-driven wire-offs that improve total completed experiences. The framework also demonstrates how Prophet-based baseline forecasting and hyperparameter search support robust, scalable decision-making in dynamic vendor ecosystems, with practical benefits for customer satisfaction and business performance.

Abstract

eBay partners with external vendors, which allows customers to freely select a vendor to complete their eBay experiences. However, vendor outages can hinder customer experiences. Consequently, eBay can disable a problematic vendor to prevent customer loss. Disabling the vendor too late risks losing customers willing to switch to other vendors, while disabling it too early risks losing those unwilling to switch. In this paper, we propose a data-driven solution to answer whether eBay should disable a problematic vendor and when to disable it. Our solution involves forecasting customer behavior. First, we use a multiplicative seasonality model to represent behavior if all vendors are fully functioning. Next, we use a Monte Carlo simulation to represent behavior if the problematic vendor remains enabled. Finally, we use a linear model to represent behavior if the vendor is disabled. By comparing these forecasts, we determine the optimal time for eBay to disable the problematic vendor.

Figures (8)

• Figure 1: Real phenomena seen during an incident in which a problematic vendor was disabled. There are several features to note: (1) as the volume of experiences completed with the problematic vendor (blue) decreases, those completed with other vendors (orange) increases; (2) the increase in experiences completed with other vendors is delayed in time with respect to the decrease in volume for the problematic vendor – it begins after the problematic vendor's volume starts decreasing and continues after the problematic vendor's volume stagnates; (3) we see an increase in experiences completed with other vendors after the problematic vendor is disabled (red dashed line). There are natural customer decisions corresponding to these phenomena: (1) as the problematic vendor's outage worsens, customers tend to migrate and complete their experiences with another vendor; (2) customers may retry with the problematic vendor several times before deciding to switch to another vendor, and these retries take a finite amount of time; (3) once the problematic vendor is disabled, more customers are willing to use a different vendor. We model all these effects in this paper.
• Figure 2: A realistic situation that strongly suggests the use of a multiplicative seasonality model. We plot the volume over time of a vendor (Top). We perform an additive seasonal decomposition on the volume time-series to extract the individual seasonal component time-series (Middle) and trend component time-series (Bottom). We connect the approximate local maxima of the seasonal component time-series with a red dashed line and the approximate local minima with a green dashed line. We indicate the approximate seasonal fluctuation amplitudes with dashed black lines. We see these amplitudes and trend decrease throughout time. Therefore, we use a multiplicative seasonality model to represent this situation, as opposed to an additive seasonality model.
• Figure 3: Decision flowchart showing the possible decisions a customer can make depending on whether they succeed with using the problematic vendor $n_0$ on their initial experience. Here, $m$ is initialized to $-10$. The Monte Carlo simulation, based on this flowchart, projects the ultimate decision (listed in a red or green box) a customer makes. The integer part of the final value of the index $m$ represents the time taken for the customer to make their decision.
• Figure 4: The proportion of the problematic vendor's baseline volume which contributed to the additional volume for all other enabled vendors (left) and its corresponding auto-correlation function (right). When performing an Augmented Dickey-Fuller test on this ratio over time, we get a test statistic of -10.33 which corresponds to a $p$-value of $2.82 \times 10^{-18}$, suggesting that this series is stationary. We can confirm this by plotting the auto-correlation function, which rapidly decays.
• Figure 5: Forecasting the baseline volume for a sample vendor using the multiplicative weekly seasonality model. We plot the true baseline volume in blue and our predicted baseline volume in orange.