DIAGPaper: Diagnosing Valid and Specific Weaknesses in Scientific Papers via Multi-Agent Reasoning
Zhuoyang Zou, Abolfazl Ansari, Delvin Ce Zhang, Dongwon Lee, Wenpeng Yin
TL;DR
DIAGPaper introduces a three-module, multi-agent framework for diagnosing weaknesses in scientific papers: Customizer decomposes evaluation into criterion-specific reviewer agents, Rebuttal enables adversarial author critiques to validate weaknesses, and Prioritizer ranks surviving weaknesses by a data-driven severity score. The approach is grounded in human-like review practices and yields top-K weaknesses with high validity and paper-specificity, outperforming both general-purpose LLMs and prior multi-agent systems on AAAR and ReviewCritique benchmarks. Key innovations include explicit criteria modeling, a multi-round paper-grounded rebuttal loop, and a meta-review-informed severity metric incorporating $\mathrm{Imp}^{c} = \frac{f^{c}_{\mathrm{meta}}}{f^{c}_{\mathrm{ind}}}$ and a weighted combination $s_w = \alpha \mathrm{Imp}^{c} + \beta \mathrm{valid}_w + (1-\alpha-\beta) \mathrm{evid}_w$ with $\alpha=0.5$, $\beta=0.3$. The results show strong cross-LLM generalization, substantial reductions of invalid weaknesses through rebuttal, and practical utility via prioritized output, with open-source models approaching closed-source baselines. This framework advances automated peer-review tooling by offering validity-oriented, paper-specific, and user-ready diagnostic feedback suitable for deployment in research communities.
Abstract
Paper weakness identification using single-agent or multi-agent LLMs has attracted increasing attention, yet existing approaches exhibit key limitations. Many multi-agent systems simulate human roles at a surface level, missing the underlying criteria that lead experts to assess complementary intellectual aspects of a paper. Moreover, prior methods implicitly assume identified weaknesses are valid, ignoring reviewer bias, misunderstanding, and the critical role of author rebuttals in validating review quality. Finally, most systems output unranked weakness lists, rather than prioritizing the most consequential issues for users. In this work, we propose DIAGPaper, a novel multi-agent framework that addresses these challenges through three tightly integrated modules. The customizer module simulates human-defined review criteria and instantiates multiple reviewer agents with criterion-specific expertise. The rebuttal module introduces author agents that engage in structured debate with reviewer agents to validate and refine proposed weaknesses. The prioritizer module learns from large-scale human review practices to assess the severity of validated weaknesses and surfaces the top-K severest ones to users. Experiments on two benchmarks, AAAR and ReviewCritique, demonstrate that DIAGPaper substantially outperforms existing methods by producing more valid and more paper-specific weaknesses, while presenting them in a user-oriented, prioritized manner.