PaperAsk: A Benchmark for Reliability Evaluation of LLMs in Paper Search and Reading

TL;DR

PaperAsk tackles the reliability problem of LLMs in scholarly workflows by evaluating three frontier models (GPT-4o, GPT-5, Gemini-2.5 Flash) on four tasks via realistic web interfaces. The study finds widespread reliability failures across citation retrieval, content extraction, paper discovery, and claim verification, driven by context pollution from broad retrieval and mismatch between sources and prompts. API-access experiments show substantial reliability gains when full-text content is provided, highlighting deployment constraints. To enable practical deployment, the authors train a lightweight reliability classifier achieving 96% accuracy to filter unreliable outputs, and they offer a reproducible benchmark framework for ongoing evaluation.

Abstract

Large Language Models (LLMs) increasingly serve as research assistants, yet their reliability in scholarly tasks remains under-evaluated. In this work, we introduce PaperAsk, a benchmark that systematically evaluates LLMs across four key research tasks: citation retrieval, content extraction, paper discovery, and claim verification. We evaluate GPT-4o, GPT-5, and Gemini-2.5-Flash under realistic usage conditions-via web interfaces where search operations are opaque to the user. Through controlled experiments, we find consistent reliability failures: citation retrieval fails in 48-98% of multi-reference queries, section-specific content extraction fails in 72-91% of cases, and topical paper discovery yields F1 scores below 0.32, missing over 60% of relevant literature. Further human analysis attributes these failures to the uncontrolled expansion of retrieved context and the tendency of LLMs to prioritize semantically relevant text over task instructions. Across basic tasks, the LLMs display distinct failure behaviors: ChatGPT often withholds responses rather than risk errors, whereas Gemini produces fluent but fabricated answers. To address these issues, we develop lightweight reliability classifiers trained on PaperAsk data to identify unreliable outputs. PaperAsk provides a reproducible and diagnostic framework for advancing the reliability evaluation of LLM-based scholarly assistance systems.

Figures (6)

• Figure 1: Overview of PaperAsk (4 tasks covering 7 disciplines). An example failure case is shown on the right.
• Figure 2: Distribution of papers across scientific fields.
• Figure 3: Citation retrieval failure breakdown. Stacked bars show three failure types: incomplete responses (returning fewer citations than requested), incorrect metadata (mismatched titles/authors/arXiv IDs), and irrelevant citations (returning entries not from the arXiv source).
• Figure 4: Claim verification failure type distribution across query sizes. Three failure types: not found (LLMs claims inability to locate provided paper), refusal (LLMs refuses to verify due to uncertainty), and incorrect verdict (wrong classification of claim).
• Figure 5: Agent performance on content extraction comparing GPT-5 (with high/minimal reasoning) and GPT-4o (non-reasoning) across two tool configurations: OpenAI's integrated search versus custom web search.