LimAgents: Multi-Agent LLMs for Generating Research Limitations
Ibrahim Al Azher, Zhishuai Guo, Hamed Alhoori
TL;DR
LimAgents introduces a multi-agent LLM framework for generating substantive research limitations by decomposing the task into explicit extraction, implicit analysis, peer-review insight, and literature-grounded context, with Judge and Master for synthesis. Ground-truth construction combines author-stated limitations with OpenReview feedback, and a pointwise LLM-based evaluation measures coverage beyond traditional n-gram metrics. Experiments show substantial gains over zero-shot baselines, with model-dependent optimal configurations (e.g., 3-Agent Llama 3 8B for smaller models and 4-Agent GPT-4o mini for larger models) achieving up to +15.51 percentage points in Ground Truth Coverage. The work demonstrates that agent-based decomposition enhances the depth and breadth of limitations, supporting more transparent and rigorous critical appraisal of scientific work, while also revealing trade-offs between coverage and quality when applying iterative feedback.
Abstract
Identifying and articulating limitations is essential for transparent and rigorous scientific research. However, zero-shot large language models (LLMs) approach often produce superficial or general limitation statements (e.g., dataset bias or generalizability). They usually repeat limitations reported by authors without looking at deeper methodological issues and contextual gaps. This problem is made worse because many authors disclose only partial or trivial limitations. We propose LimAgents, a multi-agent LLM framework for generating substantive limitations. LimAgents integrates OpenReview comments and author-stated limitations to provide stronger ground truth. It also uses cited and citing papers to capture broader contextual weaknesses. In this setup, different agents have specific roles as sequential role: some extract explicit limitations, others analyze methodological gaps, some simulate the viewpoint of a peer reviewer, and a citation agent places the work within the larger body of literature. A Judge agent refines their outputs, and a Master agent consolidates them into a clear set. This structure allows for systematic identification of explicit, implicit, peer review-focused, and literature-informed limitations. Moreover, traditional NLP metrics like BLEU, ROUGE, and cosine similarity rely heavily on n-gram or embedding overlap. They often overlook semantically similar limitations. To address this, we introduce a pointwise evaluation protocol that uses an LLM-as-a-Judge to measure coverage more accurately. Experiments show that LimAgents substantially improve performance. The RAG + multi-agent GPT-4o mini configuration achieves a +15.51% coverage gain over zero-shot baselines, while the Llama 3 8B multi-agent setup yields a +4.41% improvement.