Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation

Pujun Zheng, Jiacheng Yao, Jinquan Zheng, Chenyang Gu, Guoxiu He, Jiawei Liu, Yong Huang, Tianrui Guo, Wei Lu

Abstract

Large language models (LLMs) are currently applied to scientific paper evaluation by assigning an absolute score to each paper independently. However, since score scales vary across conferences, time periods, and evaluation criteria, models trained on absolute scores are prone to fitting narrow, context-specific rules rather than developing robust scholarly judgment. To overcome this limitation, we propose shifting paper evaluation from isolated scoring to collaborative ranking. In particular, we design \textbf{C}omparison-\textbf{N}ative framework for \textbf{P}aper \textbf{E}valuation (\textbf{CNPE}), integrating comparison into both data construction and model learning. We first propose a graph-based similarity ranking algorithm to facilitate the sampling of more informative and discriminative paper pairs from a collection. We then enhance relative quality judgment through supervised fine-tuning and reinforcement learning with comparison-based rewards. At inference, the model performs pairwise comparisons over sampled paper pairs and aggregates these preference signals into a global relative quality ranking. Experimental results demonstrate that our framework achieves an average relative improvement of \textbf{21.8\%} over the strong baseline DeepReview-14B, while exhibiting robust generalization to five previously unseen datasets. \href{https://github.com/ECNU-Text-Computing/ComparisonReview}{Code}.

From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation

Abstract

Large language models (LLMs) are currently applied to scientific paper evaluation by assigning an absolute score to each paper independently. However, since score scales vary across conferences, time periods, and evaluation criteria, models trained on absolute scores are prone to fitting narrow, context-specific rules rather than developing robust scholarly judgment. To overcome this limitation, we propose shifting paper evaluation from isolated scoring to collaborative ranking. In particular, we design \textbf{C}omparison-\textbf{N}ative framework for \textbf{P}aper \textbf{E}valuation (\textbf{CNPE}), integrating comparison into both data construction and model learning. We first propose a graph-based similarity ranking algorithm to facilitate the sampling of more informative and discriminative paper pairs from a collection. We then enhance relative quality judgment through supervised fine-tuning and reinforcement learning with comparison-based rewards. At inference, the model performs pairwise comparisons over sampled paper pairs and aggregates these preference signals into a global relative quality ranking. Experimental results demonstrate that our framework achieves an average relative improvement of \textbf{21.8\%} over the strong baseline DeepReview-14B, while exhibiting robust generalization to five previously unseen datasets. \href{https://github.com/ECNU-Text-Computing/ComparisonReview}{Code}.
Paper Structure (64 sections, 9 equations, 7 figures, 12 tables, 1 algorithm)

This paper contains 64 sections, 9 equations, 7 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Agent‑based Assessment Systems
  4. Training-based Review Models
  5. Comparison-Based Evaluation Methods
  6. Methodology
  7. Overview
  8. Pair Sampling
  9. Domain-aware Pair Ranking
  10. Sampling Strategy
  11. Training
  12. Training Pairs Filtering
  13. Cold-Start SFT and RLVR
  14. Inference
  15. Inference Pairs Filtering
  16. ...and 49 more sections

Figures (7)

  • Figure 1: Comparison of current methods (top) and our approach (bottom) across data (left), training (middle), and inference (right).
  • Figure 2: Overview of the framework. During training (bottom left), pair sampling (top left) provides sample pairs for learning quality judgment (top right). At inference (bottom right), pair sampling supplies pairs for quality judgment by the trained model, followed by ranking and decision based on preference.
  • Figure 3: Parameter analysis. Panels (a) and (b) illustrate the relationships of $n$ and $\alpha$ with Spearman’s $\rho$ (visualized in logarithmic scale); panel (c) shows their combined effects in a heatmap.
  • Figure 4: Generalization on previously unseen dataset. The differences between all groups are statistically significant.
  • Figure 5: Positional bias. Panel (a) shows the performance of the original Qwen2.5‑7B‑Instruct model; panel (b) shows the performance after SFT and RL.
  • ...and 2 more figures