Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Impact Market to Save Conference Peer Review: Decoupling Dissemination and Credentialing

Karthikeyan Sankaralingam

TL;DR

<3-5 sentence high-level summary> The paper argues that top-tier conferences conflate dissemination with credentialing, producing a damaging reviewer lottery and misaligned incentives. It proposes the Impact Market (IM), a three-phase architecture that (i) disseminates all sound work via Phase 1 publication, (ii) creates a scarce prestige signal through Phase 2 investment (Net Invested Score, NIS), and (iii) calibrates predictions against a manipulation-resistant MVIS in Phase 3 to update Investor Ratings. Through discrete-binning and agent-based simulations, the authors show that allowing expert agency and conviction betting substantially increases the retrieval of high-impact papers and makes bad actors pay a cost over time. A shadow-deployment pathway and comprehensive governance considerations are provided to validate and iteratively refine the approach before broad adoption.

Abstract

Top-tier academic conferences are failing under the strain of two irreconcilable roles: (1) rapid dissemination of all sound research and (2) scarce credentialing for prestige and career advancement. This conflict has created a reviewer roulette and anonymous tribunal model - a zero-cost attack system - characterized by high-stakes subjectivity, turf wars, and the arbitrary rejection of sound research (the equivalence class problem). We propose the Impact Market (IM), a novel three-phase system that decouples publication from prestige. Phase 1 (Publication): All sound and rigorous papers are accepted via a PC review, solving the "equivalence class" problem. Phase 2 (Investment): An immediate, scarce prestige signal is created via a futures market. Senior community members invest tokens into published papers, creating a transparent, crowdsourced Net Invested Score (NIS). Phase 3 (Calibration): A 3-year lookback mechanism validates these investments against a manipulation-resistant Multi-Vector Impact Score (MVIS). This MVIS adjusts each investor's future influence (their Investor Rating), imposing a quantifiable cost on bad actors and rewarding accurate speculation. The IM model replaces a hidden, zero-cost attack system with a transparent, accountable, and data-driven market that aligns immediate credentialing with long-term, validated impact. Agent-based simulations demonstrate that while a passive market matches current protocols in low-skill environments, introducing investor agency and conviction betting increases the retrieval of high-impact papers from 28% to over 85% under identical conditions, confirming that incentivized self-selection is the mechanism required to scale peer review.

The Impact Market to Save Conference Peer Review: Decoupling Dissemination and Credentialing

TL;DR

<3-5 sentence high-level summary> The paper argues that top-tier conferences conflate dissemination with credentialing, producing a damaging reviewer lottery and misaligned incentives. It proposes the Impact Market (IM), a three-phase architecture that (i) disseminates all sound work via Phase 1 publication, (ii) creates a scarce prestige signal through Phase 2 investment (Net Invested Score, NIS), and (iii) calibrates predictions against a manipulation-resistant MVIS in Phase 3 to update Investor Ratings. Through discrete-binning and agent-based simulations, the authors show that allowing expert agency and conviction betting substantially increases the retrieval of high-impact papers and makes bad actors pay a cost over time. A shadow-deployment pathway and comprehensive governance considerations are provided to validate and iteratively refine the approach before broad adoption.

Abstract

Top-tier academic conferences are failing under the strain of two irreconcilable roles: (1) rapid dissemination of all sound research and (2) scarce credentialing for prestige and career advancement. This conflict has created a reviewer roulette and anonymous tribunal model - a zero-cost attack system - characterized by high-stakes subjectivity, turf wars, and the arbitrary rejection of sound research (the equivalence class problem). We propose the Impact Market (IM), a novel three-phase system that decouples publication from prestige. Phase 1 (Publication): All sound and rigorous papers are accepted via a PC review, solving the "equivalence class" problem. Phase 2 (Investment): An immediate, scarce prestige signal is created via a futures market. Senior community members invest tokens into published papers, creating a transparent, crowdsourced Net Invested Score (NIS). Phase 3 (Calibration): A 3-year lookback mechanism validates these investments against a manipulation-resistant Multi-Vector Impact Score (MVIS). This MVIS adjusts each investor's future influence (their Investor Rating), imposing a quantifiable cost on bad actors and rewarding accurate speculation. The IM model replaces a hidden, zero-cost attack system with a transparent, accountable, and data-driven market that aligns immediate credentialing with long-term, validated impact. Agent-based simulations demonstrate that while a passive market matches current protocols in low-skill environments, introducing investor agency and conviction betting increases the retrieval of high-impact papers from 28% to over 85% under identical conditions, confirming that incentivized self-selection is the mechanism required to scale peer review.

Paper Structure

This paper contains 95 sections, 4 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminaries and Framing
  3. Scope of This Proposal
  4. Fundamental assumptions
  5. The Reality of Scientific Impact: A Long-Tail Distribution.
  6. What Is Outside Scope.
  7. The Failures of the Current Peer Review Model
  8. The "Equivalence Class" Lottery
  9. The "Anonymous Tribunal": A Crisis of Malice and Competence
  10. Why Existing Solutions Fail
  11. The Core Challenge: The "Premium-Mass-Market" Failure
  12. The Impact Market (IM): A 3-Phase Architecture
  13. Phase 1: The Publication Phase (Mass-Market Dissemination)
  14. Phase 2: The Investment Phase (Premium Credentialing)
  15. Privacy Protection via Time-Delayed Transparency
  16. ...and 80 more sections

Figures (4)

  • Figure 1: Citation distribution for ISCA 2017 (55 papers, 7 years post-publication). Top: Cumulative percentage of total citations. Bottom: Individual paper citation counts (log scale).
  • Figure 2: IR distributions studied
  • Figure 3: Cumulative citation distributions for ten top-tier CS conferences (2017 proceedings, analyzed 4 years post-publication). Each subplot shows the percentage of total citations accounted for by the top-ranked papers. Across all venues, the top 10 papers (typically 18-25% of the conference) account for 40-50% of citations, and the top 20 papers account for 60-85% of citations. The consistency across diverse subfields (architecture, systems, networking, programming languages, HPC) demonstrates that impact concentration is a universal property of scientific contribution, not an artifact of specific review cultures or subfield characteristics.
  • Figure 4: Citation counts (log scale) for ten top-tier CS conferences (2017, 4 years post-publication). The logarithmic scale emphasizes the power-law structure: in every venue, citations span 2-3 orders of magnitude from the least-cited to most-cited papers, with most papers clustered in the lower range. For example, ISCA's top paper has 600+ citations while its median paper has 23; SOSP's top paper exceeds 200 citations while its bottom quartile averages fewer than 20. This wide dynamic range, combined with the clustering of most papers in the "long tail," creates large equivalence classes where papers are statistically similar in impact yet subjected to binary Accept/Reject decisions under the Current Protocol.