Evaluation Sheet for Deep Research: A Use Case for Academic Survey Writing
Israel Abebe Azime, Tadesse Destaw Belay, Atnafu Lambebo Tonja
TL;DR
The paper addresses the challenge of evaluating Deep Research tools used for academic survey writing, acknowledging risks of hallucinations and unreliable sources. It proposes an Evaluation Sheet with pillars (including coverage, hallucination, source correctness, information validity, latestness, and Google-vs-LLM comparison) and demonstrates a proof-of-concept by applying it to NLP surveys focused on African languages, comparing OpenAI and Google Deep Research outputs. The results indicate below-average performance, with notable issues in source transparency, recency, and citation quality, though the approach underscores the potential and need for standardized benchmarking. The work argues for institutional adoption of evaluation standards to improve trustworthiness, interoperability, and practical utility of agentic Deep Research tools in knowledge-intensive tasks, especially for low-resource research contexts.
Abstract
Large Language Models (LLMs) powered with argentic capabilities are able to do knowledge-intensive tasks without human involvement. A prime example of this tool is Deep research with the capability to browse the web, extract information and generate multi-page reports. In this work, we introduce an evaluation sheet that can be used for assessing the capability of Deep Research tools. In addition, we selected academic survey writing as a use case task and evaluated output reports based on the evaluation sheet we introduced. Our findings show the need to have carefully crafted evaluation standards. The evaluation done on OpenAI`s Deep Search and Google's Deep Search in generating an academic survey showed the huge gap between search engines and standalone Deep Research tools, the shortcoming in representing the targeted area.