Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Score Design for Multi-Criteria Incentivization

Anmol Kabra, Mina Karzand, Tosca Lechner, Nathan Srebro, Serena Wang

TL;DR

This work gives algorithms to design scores, which are provably minimal under mild assumptions on the structure of performance metrics, to minimize the dimensionality of scores while satisfying pareto-optimal metrics.

Abstract

We present a framework for designing scores to summarize performance metrics. Our design has two multi-criteria objectives: (1) improving on scores should improve all performance metrics, and (2) achieving pareto-optimal scores should achieve pareto-optimal metrics. We formulate our design to minimize the dimensionality of scores while satisfying the objectives. We give algorithms to design scores, which are provably minimal under mild assumptions on the structure of performance metrics. This framework draws motivation from real-world practices in hospital rating systems, where misaligned scores and performance metrics lead to unintended consequences.

Score Design for Multi-Criteria Incentivization

TL;DR

This work gives algorithms to design scores, which are provably minimal under mild assumptions on the structure of performance metrics, to minimize the dimensionality of scores while satisfying pareto-optimal metrics.

Abstract

We present a framework for designing scores to summarize performance metrics. Our design has two multi-criteria objectives: (1) improving on scores should improve all performance metrics, and (2) achieving pareto-optimal scores should achieve pareto-optimal metrics. We formulate our design to minimize the dimensionality of scores while satisfying the objectives. We give algorithms to design scores, which are provably minimal under mild assumptions on the structure of performance metrics. This framework draws motivation from real-world practices in hospital rating systems, where misaligned scores and performance metrics lead to unintended consequences.
Paper Structure (44 sections, 48 theorems, 47 equations, 6 figures, 2 tables, 9 algorithms)

This paper contains 44 sections, 48 theorems, 47 equations, 6 figures, 2 tables, 9 algorithms.

Table of Contents

  1. Introduction
  2. Designing surrogate scores from performance metrics
  3. Our contributions
  4. Related work
  5. Notation
  6. Organization
  7. Minimal design problem for improvement objective
  8. Design proposal for improvement objective
  9. Geometry of metrics dictates succinctness of scores
  10. Proposed design is minimal
  11. Minimal design problem for optimality objective
  12. Design proposal for optimality objective
  13. Discussion of minimality of proposed design
  14. Minimal design problem for both objectives simultaneously
  15. Algorithms to compute matrix ranks
  16. ...and 29 more sections

Key Result

Theorem 2.2

Let columns of $\bfZ$ be an orthonormal basis of linear subspace $\calL$ associated with $\mathop{\mathrm{\mathrm{aff}}}\nolimits(\calF)$, and let $r = \dim \mathop{\mathrm{\mathrm{aff}}}\nolimits(\calF)$. For each design restriction, there exists $S : \calF \to \bbR^k$, designed using alg:1__design

Figures (6)

  • Figure 1: To design scores for two metrics ($\calF \subseteq \bbR^2$), we can inspect the correlation between metrics---the correlation dictates the succinctness of $S : \calF \to \bbR^k$ for satisfying improvement.
  • Figure 2: Side and top views of cones $\calK_Z$ (red) generated by rows of $\bfZ$, whose columns are orthonormal basis of 3-dimensional subspace $\calL$. As $\mathsf{CSR}$ and $\mathsf{CGR}$ require generating$\calK_Z$ with $\calK_V$, the matrix ranks depend on the number of extreme rays of $\calK_Z$, which can be much higher than $\dim \mathop{\mathrm{\mathrm{aff}}}\nolimits(\calF) = 3$. On the other hand, $\mathsf{CR}$ only requires enclosing$\calK_Z$ with $\calK_V$; and so is independent of the number of extreme rays.
  • Figure 3: Examples of $\calF \subseteq \bbR^2$. The left three have empty relative interior, whereas the right two have non-empty relative interior with respect to $\mathop{\mathrm{\mathrm{aff}}}\nolimits(\calF)$, which is lightly shaded.
  • Figure 4: Decomposing non-pointed cones $\calK_W$ into lineality space $\mathop{\mathrm{\mathrm{lin}}}\nolimits(\calK_W)$ and pointed $\calK_P$.
  • Figure 5: A 5-dimensional non-pointed cone $\calK_Z$ with two orthogonal components: a 2-dimensional linear subspace, and a 3-dimensional "square" pointed cone.
  • ...and 1 more figures

Theorems & Definitions (105)

  • Definition 2.1
  • Theorem 2.2
  • Example 2.3: Two correlated metrics$\implies$choose either for score design
  • Example 2.4: Two anti-correlated metrics$\implies$must choose both for score design
  • Example 2.5: Restriction with monotonicity$\implies$higher dimensionality
  • Remark 2.6: Competing metric improvement directions$\implies$higher dimensionality under Res-CS
  • Theorem 2.7
  • Remark 2.8
  • Remark 2.9: Choice of affine subspace and orthonormal basis
  • Definition 3.1
  • ...and 95 more