Optimal Scoring Rule Design under Partial Knowledge

TL;DR

This work addresses designing optimal proper scoring rules when the principal only knows a set of possible agent information structures, formalizing a max–min objective over scoring rules and information structures. By leveraging Savage's representation, the authors convert the problem into convex-function design, where the agent's information gain under a scoring rule equals the Jensen gap $\mathbb{E}[H(X)]-H(\mathbb{E}X)$. They establish that with known prior, a $v$-shaped convex $H$ remains optimal; for finite collections, an LP yields a piecewise-linear optimum; for many infinite collections, an FPTAS achieves near-optimality via $\epsilon$-covers and earth-mover distance arguments. Simulations show piecewise-linear rules perform well under strong signal–state correlation, approaching log scoring as correlation weakens, while the classic $v$-shaped rule can underperform in partial-knowledge settings. Together, these results provide practical scoring-rule design methods that robustly incentivize information acquisition under uncertainty, with broad implications for crowdsourcing, peer review, and predictive elicitation.

Abstract

This paper studies the design of optimal proper scoring rules when the principal has partial knowledge of an agent's signal distribution. Recent work characterizes the proper scoring rules that maximize the increase of an agent's payoff when the agent chooses to access a costly signal to refine a posterior belief from her prior prediction, under the assumption that the agent's signal distribution is fully known to the principal. In our setting, the principal only knows about a set of distributions where the agent's signal distribution belongs. We formulate the scoring rule design problem as a max-min optimization that maximizes the worst-case increase in payoff across the set of distributions. We propose an efficient algorithm to compute an optimal scoring rule when the set of distributions is finite, and devise a fully polynomial-time approximation scheme that accommodates various infinite sets of distributions. We further remark that widely used scoring rules, such as the quadratic and log rules, as well as previously identified optimal scoring rules under full knowledge, can be far from optimal in our partial knowledge settings.

Figures (10)

• Figure 1: The optimal scoring rules for the homogeneous prior setting. With \ref{['thm:uniprior']}, the dashed lines are the optimal $v$-shaped convex functions for three different priors ($0.7, 0.5$, and $0.2$) in the ex-post setting, and the solid ones are in the ex-ante setting.
• Figure 2: The optimal piecewise linear functions for a finite collection of information structures $\mathcal{P}_{\rho, N} = \{(\pi, \sigma)\in \mathcal{P}_\rho : N\pi\in \mathbb{N}\}$ with $\rho = 0.1$ and $N = 10$ (\ref{['sec:sim']}). In both ex-post and ex-ante setting, the vertex of the function is at the prior of the collection of information structures which is aligned with our intuition in \ref{['sec:infinite']} which suggests maximizing curvature at the prior.
• Figure 3: Associated convex functions
• Figure 4: Information gain with $\rho = 0.25$.
• Figure 5: Information gain with $\rho = 0.025$.

Theorems & Definitions (34)

• Definition 2.1
• Definition 2.2
• Example 2.3
• Example 2.4
• Example 2.5
• Theorem 2.6: mccarthy1956measuressavage1971elicitation
• Lemma 2.7
• Lemma 2.8
• Proposition 3.1: singleton
• Theorem 3.2: homogeneous prior