Diversification as Risk Minimization

TL;DR

This work tackles robustness within a single query by recognizing that ambiguous queries induce multiple user intents, and existing diversification metrics often fail to protect minority intents. It introduces VRisk, a CVaR-inspired tail-risk metric that measures the expected loss for the worst $\beta$-fraction of intents, and VRisker, a greedy re-ranker with $(1-1/e)$ guarantees for modular metrics and data-dependent guarantees for non-modular metrics. Empirical results on INTENT-2, TREC Web 2012, and MovieLens show VRisker can reduce worst-case intent failures by up to 33% with only about a 2% drop in average relevance, outperforming standard diversification approaches in robustness. The framework provides tunable safety through $\beta$ and baseline targets, offering a principled way to balance robustness and utility in both search and recommender settings, with limitations related to intent estimation and mutual exclusivity of intents. Overall, the paper advances intent-aware diversification by connecting it to risk minimization and providing scalable, theoretically grounded optimization methods.

Abstract

Users tend to remember failures of a search session more than its many successes. This observation has led to work on search robustness, where systems are penalized if they perform very poorly on some queries. However, this principle of robustness has been overlooked within a single query. An ambiguous or underspecified query (e.g., ``jaguar'') can have several user intents, where popular intents often dominate the ranking, leaving users with minority intents unsatisfied. Although the diversification literature has long recognized this issue, existing metrics only model the average relevance across intents and provide no robustness guarantees. More surprisingly, we show theoretically and empirically that many well-known diversification algorithms are no more robust than a naive, non-diversified algorithm. To address this critical gap, we propose to frame diversification as a risk-minimization problem. We introduce VRisk, which measures the expected risk faced by the least-served fraction of intents in a query. Optimizing VRisk produces a robust ranking, reducing the likelihood of poor user experiences. We then propose VRisker, a fast greedy re-ranker with provable approximation guarantees. Finally, experiments on NTCIR INTENT-2, TREC Web 2012, and MovieLens show the vulnerability of existing methods. VRisker reduces worst-case intent failures by up to 33% with a minimal 2% drop in average performance.

Figures (16)

• Figure 1: Illustration of VRisk. Bar height shows per-intent loss $\ell(R_{k}, q, c)$ and the bar width shows the intent probability $\Pr\bigl(c|q\bigr)$. 5 intents are sorted by loss (worst $\to$ best). VRisk takes the average of , the worst-$\beta$ intent probability mass.
• Figure 2: VRisker is robust across base metrics $V$. The plots show how different methods perform on varying base metrics $V$, tested on NTCIR INTENT-2 (left), TREC Web 2012 (middle), and MovieLens 32M (right). The top figures evaluate risk, $\Delta V_\text{Risk}$ (smaller is more robust), which measures the expected loss of the worst $\beta$-fraction of intents. The bottom figures evaluate the average performance, $\Delta V_\text{IW}$ and $\Delta V_\text{std}$ (larger is better). All values are relative to Naive = 100%. ($k=10$, $\beta=0.10$)
• Figure 3: VRisk/VRisker's pessimism is controllable via $\beta$. The plots show results on VRisk and $V_{\text{std}}$ (= $V_{\text{IW}}$).
• Figure 4: VRisker is robust to ranking length.
• Figure 5: The results are consistent when intent probabilities with noise added. VRisk and $V_{\text{std}}$ shown with raw values.

Theorems & Definitions (5)

• Example 1
• Proposition 1
• Proposition 2: NP-hardness
• Theorem 1: $(1-\!1/e)$ Optimality Guarantee
• Theorem 2: nDCG-Risk Approximation