Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ranking with Multiple Objectives

Nikhil R. Devanur, Sivakanth Gopi

TL;DR

A general class of problems where each result gets a different score corresponding to each objective, and the results of a ranking are aggregated by taking, for each objective, a weighted sum of the scores in the order of the ranking, and an arbitrary concave function of the aggregates is maximized.

Abstract

In search and advertisement ranking, it is often required to simultaneously maximize multiple objectives. For example, the objectives can correspond to multiple intents of a search query, or in the context of advertising, they can be relevance and revenue. It is important to efficiently find rankings which strike a good balance between such objectives. Motivated by such applications, we formulate a general class of problems where - each result gets a different score corresponding to each objective, - the results of a ranking are aggregated by taking, for each objective, a weighted sum of the scores in the order of the ranking, and - an arbitrary concave function of the aggregates is maximized. Combining the aggregates using a concave function will naturally lead to more balanced outcomes. We give an approximation algorithm in a bicriteria/resource augmentation setting: the algorithm with a slight advantage does as well as the optimum. In particular, if the aggregation step is just the sum of the top k results, then the algorithm outputs k + 1 results which do as well the as the optimal top k results. We show how this approach helps with balancing different objectives via simulations on synthetic data as well as on real data from LinkedIn.

Ranking with Multiple Objectives

TL;DR

A general class of problems where each result gets a different score corresponding to each objective, and the results of a ranking are aggregated by taking, for each objective, a weighted sum of the scores in the order of the ranking, and an arbitrary concave function of the aggregates is maximized.

Abstract

In search and advertisement ranking, it is often required to simultaneously maximize multiple objectives. For example, the objectives can correspond to multiple intents of a search query, or in the context of advertising, they can be relevance and revenue. It is important to efficiently find rankings which strike a good balance between such objectives. Motivated by such applications, we formulate a general class of problems where - each result gets a different score corresponding to each objective, - the results of a ranking are aggregated by taking, for each objective, a weighted sum of the scores in the order of the ranking, and - an arbitrary concave function of the aggregates is maximized. Combining the aggregates using a concave function will naturally lead to more balanced outcomes. We give an approximation algorithm in a bicriteria/resource augmentation setting: the algorithm with a slight advantage does as well as the optimum. In particular, if the aggregation step is just the sum of the top k results, then the algorithm outputs k + 1 results which do as well the as the optimal top k results. We show how this approach helps with balancing different objectives via simulations on synthetic data as well as on real data from LinkedIn.

Paper Structure

This paper contains 12 sections, 9 theorems, 48 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Main Results
  3. Related Work
  4. Organization:
  5. A single instance of ranking with multiple objectives
  6. Multiple instances of ranking with global aggregation
  7. Experiments
  8. Synthetic Data
  9. Real Data
  10. Running time of binary search in Proposition \ref{['prop:binary_search_OPT']}
  11. Getting $O(n\log n\log B)$ running time.
  12. Getting strongly polynomial randomized $O(n\log^2 n)$ running time.

Key Result

theorem 2

Suppose $f(\alpha,\beta)$ is a concave function over the range $\alpha,\beta\ge 0$ and $f$ is strictly increasing in each coordinate in that range. Given an instance of $\operatorname{\mathsf{Rank}}({\mathbf a},{\mathbf b},{\mathbf w},f)$, there is an algorithm that runs in $O(n\log^2 n)$ timeThis i

Figures (8)

  • Figure 1: Scatter plot of NDCGs for two different objectives, $A$ and $B$, on real data.
  • Figure 2: The positive quadrant $\mathcal{Q}^{++}$ is divided into regions $A_i$ and rays $R_i$ based on the values of the subgradient $\partial \operatorname{\mathsf{cs}}^*(p{\mathbf a}+q{\mathbf b})$.
  • Figure 4: The positive quadrant $\mathcal{Q}^{++}$ is divided into regions $A_j$ and rays $R_j$ based on the values of the subgradients $\partial cs^*_{{\mathbf w}_i}\left((p+p_i){\mathbf a}_i+(q+q_i){\mathbf b}_i\right)$. One can use binary search as in the proof of Proposition \ref{['prop:binary_search_OPT']} to solve the fixed point equation (\ref{['eqn:fixedpoint_global']}).
  • Figure 5: Distribution of $a_{ij}$ and $b_{ij}$ values drawn from an anti-correlated log normal distribution.
  • Figure 6: Scatter plot of NDCGs for two different objectives, $A$ and $B$, on synthetic data.
  • ...and 3 more figures

Theorems & Definitions (27)

  • definition 1: $\operatorname{\mathsf{Rank}}({\mathbf a},{\mathbf b},{\mathbf w},f)$
  • theorem 2
  • corollary 3
  • proposition 4
  • proof
  • lemma 5
  • proof
  • remark 6
  • remark 7
  • proposition 8: Binary search to find $\mathsf{OPT}$
  • ...and 17 more