CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories
Yijia Xiao, Runhui Wang, Luyang Kong, Davor Golac, Wei Wang
TL;DR
CSR-Bench introduces a benchmark and a multi-agent framework (CSR-Agents) to automate end-to-end deployment of computer science research repositories. The approach uses a Docker-based isolation environment and five cooperating LLM agents to draft, execute, debug, and improve deployment commands guided by repository READMEs, issue databases, and web search. Evaluations across multiple foundation models show that inclusion of additional agents and retrieval tools improves deployment success, but complex tasks like training and inference remain challenging for autonomous execution. The work provides a reproducible benchmark and CI/CD-like workflow for CS research deployments, highlighting both the potential of LLM-driven automation and the remaining research gaps for full autonomy.
Abstract
The increasing complexity of computer science research projects demands more effective tools for deploying code repositories. Large Language Models (LLMs), such as Anthropic Claude and Meta Llama, have demonstrated significant advancements across various fields of computer science research, including the automation of diverse software engineering tasks. To evaluate the effectiveness of LLMs in handling complex code development tasks of research projects, particularly for NLP/CV/AI/ML/DM topics, we introduce CSR-Bench, a benchmark for Computer Science Research projects. This benchmark assesses LLMs from various aspects including accuracy, efficiency, and deployment script quality, aiming to explore their potential in conducting computer science research autonomously. We also introduce a novel framework, CSR-Agents, that utilizes multiple LLM agents to automate the deployment of GitHub code repositories of computer science research projects. Specifically, by checking instructions from markdown files and interpreting repository structures, the model generates and iteratively improves bash commands that set up the experimental environments and deploy the code to conduct research tasks. Preliminary results from CSR-Bench indicate that LLM agents can significantly enhance the workflow of repository deployment, thereby boosting developer productivity and improving the management of developmental workflows.