A Cramér-von Mises Approach to Incentivizing Truthful Data Sharing

TL;DR

The paper develops CvM-inspired, two-sample test–based losses to incentivize truthful data submissions in data marketplaces without strong distributional assumptions. It proves that truthful reporting forms a Nash equilibrium in a Bayesian setting and yields an ε-approximate equilibrium in a prior-agnostic setting, while also incentivizing larger, higher-quality submissions. The authors instantiate the mechanism in three data-sharing problems—data purchasing, data collection marketplaces, and federated learning—and provide theoretical guarantees plus empirical validation on synthetic, text, and image data. The approach offers a practical, distribution-agnostic framework for robust data sharing in the presence of fabrication and strategic behavior.

Abstract

Modern data marketplaces and data sharing consortia increasingly rely on incentive mechanisms to encourage agents to contribute data. However, schemes that reward agents based on the quantity of submitted data are vulnerable to manipulation, as agents may submit fabricated or low-quality data to inflate their rewards. Prior work has proposed comparing each agent's data against others' to promote honesty: when others contribute genuine data, the best way to minimize discrepancy is to do the same. Yet prior implementations of this idea rely on very strong assumptions about the data distribution (e.g. Gaussian), limiting their applicability. In this work, we develop reward mechanisms based on a novel, two-sample test inspired by the Cramér-von Mises statistic. Our methods strictly incentivize agents to submit more genuine data, while disincentivizing data fabrication and other types of untruthful reporting. We establish that truthful reporting constitutes a (possibly approximate) Nash equilibrium in both Bayesian and prior-agnostic settings. We theoretically instantiate our method in three canonical data sharing problems and show that it relaxes key assumptions made by prior work. Empirically, we demonstrate that our mechanism incentivizes truthful data sharing via simulations and on real-world language and image data.

Figures (3)

• Figure 1: Subfigure (\ref{['fig:two-sample-cvm']}) shows the empirical CDFs (ECDF) for two datasets $X=\left\{X_1,\ldots,X_n\right\}$, $Y=\left\{Y_1,\ldots,Y_m\right\}$. The gray lines are the differences between the two curves at each point in $\left( X_1,\ldots,X_n, Y_1,\ldots,Y_m \right)$, and are used to calculate the two-sample CvM test in \ref{['eq:cvmtest']}. Subfigure (\ref{['fig:ecdf-vs-cond-ex']}) replaces $F_Y(t)$ with $\mathbb{E}[F_Y(t)|X]$ which can be thought of as the best approximation to $F_{Y}(t)$ based on having seen $X$.
• Figure 2: (\ref{['fig:beta-bern-experiments']}): Losses when submitting truthfully, adding ${\bf{\rm Bern}}\,(1/2)$ samples, and adding ${\bf{\rm Bern}}\,(\tilde{p})$ samples in the beta-Bernoulli experiment. (\ref{['fig:normal-normal-experiments']}): Losses when submitting truthfully and adding fabricated data between adjacent pairs of true data points in the normal-normal experiment. In (\ref{['fig:normal-normal-experiments']}), the CvM bar for fabrication behavior extends to $\approx 1.6$. Losses for truthful submission in each method and subfigure are normalized to 1 (gray lines); values $<1$ indicate fabrication improves performance, $>1$ means it worsens. A truthful mechanism should yield losses above 1 for all fabrication behavior.
• Figure 3: Pictured above is an example prompt fed into Llama 3.2-1B-Instruct as part of an untruthful agent's submission function to generate fabricated text data. The agent uses their five questions drawn from the SQuAD to fabricate similar five additonal questions.

Theorems & Definitions (56)

• Definition 1
• Theorem 1
• Proposition 1
• Definition 2
• Theorem 2
• Theorem 3
• Definition 3
• Proposition 2
• proof
• Definition 4