Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WarrantScore: Modeling Warrants between Claims and Evidence for Substantiation Evaluation in Peer Reviews

Kiyotada Mori, Shohei Tanaka, Tosho Hirasawa, Tadashi Kozuno, Koichiro Yoshino, Yoshitaka Ushiku

TL;DR

WarrantScore extends SubstanScore by generating and evaluating warrants that connect claims to evidence, aiming to capture the logical inference linking components in scientific reviews. The approach uses a warrant generator to produce warrants, a binary classifier to assess acceptability, and an LLM-based judge to rate plausibility on a four-point scale, with the final score scaled to [0,1] and multiplied by review length. Empirical results on SubstanReview and RottenReview show that WarrantScore and warrant_rate achieve stronger correlations with human judgments than existing baselines, and that warrant-based evaluation maintains robustness against superficial features like review length. The work highlights the value of explicitly modeling warrants for interpretable, human-aligned peer-review evaluation, while noting limitations such as fixed warrant types and limited domain coverage, and calling for larger, more diverse annotated datasets for future validation.

Abstract

The scientific peer-review process is facing a shortage of human resources due to the rapid growth in the number of submitted papers. The use of language models to reduce the human cost of peer review has been actively explored as a potential solution to this challenge. A method has been proposed to evaluate the level of substantiation in scientific reviews in a manner that is interpretable by humans. This method extracts the core components of an argument, claims and evidence, and assesses the level of substantiation based on the proportion of claims supported by evidence. The level of substantiation refers to the extent to which claims are based on objective facts. However, when assessing the level of substantiation, simply detecting the presence or absence of supporting evidence for a claim is insufficient; it is also necessary to accurately assess the logical inference between a claim and its evidence. We propose a new evaluation metric for scientific review comments that assesses the logical inference between claims and evidence. Experimental results show that the proposed method achieves a higher correlation with human scores than conventional methods, indicating its potential to better support the efficiency of the peer-review process.

WarrantScore: Modeling Warrants between Claims and Evidence for Substantiation Evaluation in Peer Reviews

TL;DR

WarrantScore extends SubstanScore by generating and evaluating warrants that connect claims to evidence, aiming to capture the logical inference linking components in scientific reviews. The approach uses a warrant generator to produce warrants, a binary classifier to assess acceptability, and an LLM-based judge to rate plausibility on a four-point scale, with the final score scaled to [0,1] and multiplied by review length. Empirical results on SubstanReview and RottenReview show that WarrantScore and warrant_rate achieve stronger correlations with human judgments than existing baselines, and that warrant-based evaluation maintains robustness against superficial features like review length. The work highlights the value of explicitly modeling warrants for interpretable, human-aligned peer-review evaluation, while noting limitations such as fixed warrant types and limited domain coverage, and calling for larger, more diverse annotated datasets for future validation.

Abstract

The scientific peer-review process is facing a shortage of human resources due to the rapid growth in the number of submitted papers. The use of language models to reduce the human cost of peer review has been actively explored as a potential solution to this challenge. A method has been proposed to evaluate the level of substantiation in scientific reviews in a manner that is interpretable by humans. This method extracts the core components of an argument, claims and evidence, and assesses the level of substantiation based on the proportion of claims supported by evidence. The level of substantiation refers to the extent to which claims are based on objective facts. However, when assessing the level of substantiation, simply detecting the presence or absence of supporting evidence for a claim is insufficient; it is also necessary to accurately assess the logical inference between a claim and its evidence. We propose a new evaluation metric for scientific review comments that assesses the logical inference between claims and evidence. Experimental results show that the proposed method achieves a higher correlation with human scores than conventional methods, indicating its potential to better support the efficiency of the peer-review process.
Paper Structure (23 sections, 5 equations, 3 figures, 6 tables)

This paper contains 23 sections, 5 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation of Peer Review Comments
  4. Argument Mining
  5. Problem Formulation
  6. Proposed Method
  7. SubstanScore (Baseline)
  8. WarrantScore
  9. Warrant Generation and Evaluation
  10. Correlation Analysis with Human Judgments
  11. Evaluation Metrics
  12. Dataset
  13. Result
  14. Robustness Analysis of Evaluation Metrics
  15. Evaluation Metric
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Comparison between proposed peer-review evaluation metrics that evaluate claims, evidence, and warrants and baseline metrics that only evaluate claims and evidence.
  • Figure 2: Warrant-generation and evaluation pipeline using LLMs: first, an LLM (warrant generator) generates a warrant that connects a claim and evidence pair; next, a binary classification model (warrant evaluator) verifies its acceptability; and finally, another LLM (LLM-as-a-judge) evaluates the warrant on a four-point scale.
  • Figure 3: Number of words in the original peer-review and the elongated peer-review.