Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Eliciting Informative Text Evaluations with Large Language Models

Yuxuan Lu, Shengwei Xu, Yichi Zhang, Yuqing Kong, Grant Schoenebeck

TL;DR

This work introduces two mechanisms, the Generative Peer Prediction Mechanism (GPPM) and the Generative Synopsis Peer Prediction Mechanism (GSPPM), which utilize LLMs as predictors, mapping from one agent's report to a prediction of her peer's report.

Abstract

Peer prediction mechanisms motivate high-quality feedback with provable guarantees. However, current methods only apply to rather simple reports, like multiple-choice or scalar numbers. We aim to broaden these techniques to the larger domain of text-based reports, drawing on the recent developments in large language models. This vastly increases the applicability of peer prediction mechanisms as textual feedback is the norm in a large variety of feedback channels: peer reviews, e-commerce customer reviews, and comments on social media. We introduce two mechanisms, the Generative Peer Prediction Mechanism (GPPM) and the Generative Synopsis Peer Prediction Mechanism (GSPPM). These mechanisms utilize LLMs as predictors, mapping from one agent's report to a prediction of her peer's report. Theoretically, we show that when the LLM prediction is sufficiently accurate, our mechanisms can incentivize high effort and truth-telling as an (approximate) Bayesian Nash equilibrium. Empirically, we confirm the efficacy of our mechanisms through experiments conducted on two real datasets: the Yelp review dataset and the ICLR OpenReview dataset. We highlight the results that on the ICLR dataset, our mechanisms can differentiate three quality levels -- human-written reviews, GPT-4-generated reviews, and GPT-3.5-generated reviews in terms of expected scores. Additionally, GSPPM penalizes LLM-generated reviews more effectively than GPPM.

Eliciting Informative Text Evaluations with Large Language Models

TL;DR

This work introduces two mechanisms, the Generative Peer Prediction Mechanism (GPPM) and the Generative Synopsis Peer Prediction Mechanism (GSPPM), which utilize LLMs as predictors, mapping from one agent's report to a prediction of her peer's report.

Abstract

Peer prediction mechanisms motivate high-quality feedback with provable guarantees. However, current methods only apply to rather simple reports, like multiple-choice or scalar numbers. We aim to broaden these techniques to the larger domain of text-based reports, drawing on the recent developments in large language models. This vastly increases the applicability of peer prediction mechanisms as textual feedback is the norm in a large variety of feedback channels: peer reviews, e-commerce customer reviews, and comments on social media. We introduce two mechanisms, the Generative Peer Prediction Mechanism (GPPM) and the Generative Synopsis Peer Prediction Mechanism (GSPPM). These mechanisms utilize LLMs as predictors, mapping from one agent's report to a prediction of her peer's report. Theoretically, we show that when the LLM prediction is sufficiently accurate, our mechanisms can incentivize high effort and truth-telling as an (approximate) Bayesian Nash equilibrium. Empirically, we confirm the efficacy of our mechanisms through experiments conducted on two real datasets: the Yelp review dataset and the ICLR OpenReview dataset. We highlight the results that on the ICLR dataset, our mechanisms can differentiate three quality levels -- human-written reviews, GPT-4-generated reviews, and GPT-3.5-generated reviews in terms of expected scores. Additionally, GSPPM penalizes LLM-generated reviews more effectively than GPPM.
Paper Structure (67 sections, 11 theorems, 26 equations, 13 figures, 11 tables, 2 algorithms)

This paper contains 67 sections, 11 theorems, 26 equations, 13 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Our Contribution
  3. Related Work
  4. Peer prediction mechanisms
  5. Shortcuts in information elicitation
  6. LLM applications in peer review
  7. Preliminaries
  8. Model
  9. Item and Signal
  10. Reporting Strategy
  11. Effort Model
  12. Synopsis
  13. Peer Prediction Mechanism and Agents' Incentive
  14. Mechanism Design Goal
  15. miller2005eliciting's Peer Prediction Mechanism
  16. ...and 52 more sections

Key Result

Proposition 3.8

In the binary effort model, if the common prior $\pi$ and $\Pr[X_i\mid Z=z]$ are known, the above mechanism is potent.

Figures (13)

  • Figure 1: An example from our study: Two reviews of a submission at ICLR2020, the left one by a human reviewer, and the right one by GPT-4.
  • Figure 2: An Overview of Our Information Elicitation Model
  • Figure 3: An Example of the Three-Level Effort Model
  • Figure 4: An Example of the Synopsis-determined Low-effort Signals
  • Figure 5: A simplified example of a prompt for Token in academic peer review scenario. In Appendix \ref{['appendix:prompt']}, we present all exact prompts used in our experiment.
  • ...and 8 more figures

Theorems & Definitions (19)

  • Definition 3.2: Synopsis-determined Low-effort Signals
  • Definition 3.3: Synopsis-covering Low-effort Signals
  • Definition 3.4
  • Definition 3.5: Potent Mechanism
  • Definition 3.6: Log Scoring Rule (LSR)
  • Definition 3.7: Original Peer Prediction Mechanism
  • Proposition 3.8: Proposition 1 of miller2005eliciting
  • Definition 4.2: Generative Peer Prediction Mechanism (GPPM)
  • Definition 4.3: Generative Synopsis Peer Prediction Mechanism (GSPPM)
  • Proposition 4.3
  • ...and 9 more