Truthful Reporting of Competence with Minimal Verification

Abstract

Suppose you run a home exam, where students should report their own scores but can cheat freely. You can, if needed, call a limited number of students to class and verify their actual performance against their reported score. We consider the class of mechanisms where truthful reporting is a dominant strategy, and truthful agents are never penalized -- even off-equilibrium. How many students do we need to verify, in expectation, if we want to minimize the bias, i.e., the difference between agents' competence and their expected grade? When perfect verification is available, we characterize the best possible tradeoff between these requirements and provide a simple parametrized mechanism that is optimal in the class for any distribution of agents' types. When verification is noisy, the task becomes much more challenging. We show how proper scoring rules can be leveraged in different ways to construct truthful mechanisms with a good (though not necessarily optimal) tradeoff.

Figures (6)

• Figure 1: Top: Typical distribution of credit score FICO. Bottom: Typical distribution of SAT scores wikipediaSAT.
• Figure 2: Illustration of an MCV mechanism. The horizontal line represents the report $\hat{t}_i=t$, and the vertical axis plays the role of both the grade and the audit probabilities. The audit probability (red double line) is zero for reports below $\gamma$, and any such report gets $\gamma$ (green horizontal segment). The dotted diagonal line represents the identity. When $\hat{t}_i>\gamma$, audit probability is increasing, both the default (green) and truthful (dashed orange) grades match $\hat{t}_i$, and there is a fixed maximal penalty of $-\xi$ (blue dash-dots).
• Figure 3: US income distribution by tax brackets in 2013 IRS.
• Figure 4: Visualization of the Polynomial Verification mechanism with $\kappa=2, \theta=0.5$. The purple dash-dot line shows the expected grade of a truthful agent. As in Fig. \ref{['fig:MCV']}, the shaded areas show the overall verification and bias under a uniform type distribution.
• Figure 5: Efficiency curves for MCV mechanism under Limited Liability on several synthetic and real type distributions. The $\gamma$ parameter ranges from 0 to 1, with higher values to the right.

Theorems & Definitions (17)

• definition 1: Domination
• definition 2: Efficiency
• proposition 1
• theorem 1
• corollary 1
• corollary 2
• corollary 3
• definition 3
• theorem 2
• lemma 1