ResearchGPT: Benchmarking and Training LLMs for End-to-End Computer Science Research Workflows

TL;DR

The paper introduces CS-54k, a paper-grounded, end-to-end dataset for evaluating AI as a research assistant in computer science, and derives CS-4k as a rigorous end-to-end benchmark and CS-50k as a training corpus. It presents a scalable RAG-based data generation and a multi-stage quality-control pipeline to ensure factual grounding in authentic papers. Experimental results show that CS-4k reveals clear capability tiers among state-of-the-art models, and that domain-aligned training on CS-50k with supervised fine-tuning and GRPO reinforcement learning yields substantial gains, even at 7B scales, sometimes matching or surpassing larger proprietary models. The work argues that domain-specific data quality and alignment are more crucial than sheer pretraining scale and outlines a path toward multimodal, fully grounded AI researchers for scientific discovery.

Abstract

As large language models (LLMs) advance, the ultimate vision for their role in science is emerging: we could build an AI collaborator to effectively assist human beings throughout the entire scientific research process. We refer to this envisioned system as ResearchGPT. Given that scientific research progresses through multiple interdependent phases, achieving this vision requires rigorous benchmarks that evaluate the end-to-end workflow rather than isolated sub-tasks. To this end, we contribute CS-54k, a high-quality corpus of scientific Q&A pairs in computer science, built from 14k CC-licensed papers. It is constructed through a scalable, paper-grounded pipeline that combines retrieval-augmented generation (RAG) with multi-stage quality control to ensure factual grounding. From this unified corpus, we derive two complementary subsets: CS-4k, a carefully curated benchmark for evaluating AI's ability to assist scientific research, and CS-50k, a large-scale training dataset. Extensive experiments demonstrate that CS-4k stratifies state-of-the-art LLMs into distinct capability tiers. Open models trained on CS-50k with supervised training and reinforcement learning demonstrate substantial improvements. Even 7B-scale models, when properly trained, outperform many larger proprietary systems, such as GPT-4.1, GPT-4o, and Gemini 2.5 Pro. This indicates that making AI models better research assistants relies more on domain-aligned training with high-quality data than on pretraining scale or general benchmark performance. We release CS-4k and CS-50k in the hope of fostering AI systems as reliable collaborators in CS research.

Figures (8)

• Figure 1: Benchmarking Comparison on CS-4k. Left: Overall scores of selected models, illustrating clear performance differences. Right: A comparison of model size vs. score, where some proprietary model sizes are estimates blog:o3mini_vs_deepseekr1blog:claude4_vs_llama4abacha2024medec. The red arrow highlights the substantial gain from training Qwen on CS-50k. The orange and purple dashed lines indicate the scaling trends of instruction-tuned and reasoning-oriented models.
• Figure 2: Dataset construction pipeline. Top: source papers from six CS conferences are converted, chunked, and embedded. Bottom: question expansion, answer grounding, and multi-stage quality control yield approximately 50k high-quality Q&A pairs.
• Figure 3: Overall dataset statistics for CS-4k/50k combined. (a) Category coverage across 8 research workflow dimensions. (b) the distribution of difficulty scores estimated by an LLM-based scorer (histogram with KDE fit). (c) Input length variation across categories.
• Figure 4: Training and evaluation statistics with single vs. dual reward models: output length curves, evaluation reward curves, and CS-4k scores across research dimensions.
• Figure 5: Case studies on CS-4k across different research workflow dimensions.