Bridging Social Psychology and LLM Reasoning: Conflict-Aware Meta-Review Generation via Cognitive Alignment

TL;DR

This paper tackles automating meta-review generation by addressing conflict-aware reasoning among multiple reviewer opinions. It introduces the Cognitive Alignment Framework (CAF), a three-phase, dual-process architecture that initializes key information, iteratively integrates reviews with conflict detection, and applies fast-slow thinking to resolve disagreements. Empirical results demonstrate improvements in sentiment and content consistency for several LLMs and show reductions in anchoring and conformity biases, as well as a detailed case study illustrating effective conflict resolution. Limitations include dataset scope, lack of rebuttal consideration, and ethical considerations, guiding future refinements and broader validation.

Abstract

The rapid growth of scholarly submissions has overwhelmed traditional peer review systems, driving the need for intelligent automation to preserve scientific rigor. While large language models (LLMs) show promise in automating manuscript critiques, their ability to synthesize high-stakes meta-reviews, which require conflict-aware reasoning and consensus derivation, remains underdeveloped. Existing methods fail to effectively handle conflicting viewpoints within differing opinions, and often introduce additional cognitive biases, such as anchoring effects and conformity bias.To overcome these limitations, we propose the Cognitive Alignment Framework (CAF), a dual-process architecture that transforms LLMs into adaptive scientific arbitrators. By operationalizing Kahneman's dual-process theory, CAF introduces a three-step cognitive pipeline: review initialization, incremental integration, and cognitive alignment.Empirical validation shows that CAF outperforms existing LLM-based methods, with sentiment consistency gains reaching up to 19.47\% and content consistency improving by as much as 12.95\%.

Figures (8)

• Figure 1: Illustrating the "Fast Thinking” and "Slow Thinking” processes in human meta-review.
• Figure 2: The impact of different reviewers (R1, R2, R3, R4, R5) on the final score, with higher y-values indicating greater weight. $\Delta$ represents the difference in anchoring effect between GPT and human reviewers.
• Figure 3: The architecture of our proposed CAF model. It begins with Key Information Extraction, where initial reviews are collected and key elements are extracted ($\S \ref{['s1']}$). Next, the Conflict-Aware Iterative Integration phase integrates the reviews while detecting and resolving conflicts ($\S \ref{['s2']}$). Finally, the Dual-Process Cognitive Alignment stage applies both fast and slow thinking strategies to address conflicts, ultimately generating the meta-review ($\S \ref{['s3']}$).
• Figure 4: The relationship between score differences in reviews and the probability of needing cognitive reconstruction (reflection) in the meta-review process.
• Figure 5: The impact of different reviewers on the final score, with higher y-values indicating greater weight.