KOCO-BENCH: Can Large Language Models Leverage Domain Knowledge in Software Development?

TL;DR

KoCo-Bench addresses the gap in evaluating how large language models acquire and apply domain knowledge for real-world software development by pairing knowledge corpora with evaluation tasks. It introduces two tasks—domain code generation and domain knowledge understanding—across six emerging domains, 11 frameworks, and 25 projects, with a rigorous construction pipeline and multi-level requirements and tests. The empirical results show that state-of-the-art LLMs struggle with domain-specific tasks, domain specialization methods offer only modest improvements, and agent-based approaches provide the best but still insufficient performance, with challenges like forgetting and corpus-size effects. The benchmark thus provides a robust, data-grounded platform to drive development of more effective domain specialization methods and will help catalyze progress in domain-aware software engineering AI.

Abstract

Large language models (LLMs) excel at general programming but struggle with domain-specific software development, necessitating domain specialization methods for LLMs to learn and utilize domain knowledge and data. However, existing domain-specific code benchmarks cannot evaluate the effectiveness of domain specialization methods, which focus on assessing what knowledge LLMs possess rather than how they acquire and apply new knowledge, lacking explicit knowledge corpora for developing domain specialization methods. To this end, we present KOCO-BENCH, a novel benchmark designed for evaluating domain specialization methods in real-world software development. KOCO-BENCH contains 6 emerging domains with 11 software frameworks and 25 projects, featuring curated knowledge corpora alongside multi-granularity evaluation tasks including domain code generation (from function-level to project-level with rigorous test suites) and domain knowledge understanding (via multiple-choice Q&A). Unlike previous benchmarks that only provide test sets for direct evaluation, KOCO-BENCH requires acquiring and applying diverse domain knowledge (APIs, rules, constraints, etc.) from knowledge corpora to solve evaluation tasks. Our evaluations reveal that KOCO-BENCH poses significant challenges to state-of-the-art LLMs. Even with domain specialization methods (e.g., SFT, RAG, kNN-LM) applied, improvements remain marginal. Best-performing coding agent, Claude Code, achieves only 34.2%, highlighting the urgent need for more effective domain specialization methods. We release KOCO-BENCH, evaluation code, and baselines to advance further research at https://github.com/jiangxxxue/KOCO-bench.

Figures (7)

• Figure 1: An overview of KoCo-Bench.
• Figure 2: Construction pipeline of KoCo-Bench.
• Figure 3: The impact of knowledge corpus size on domain specialization performance.
• Figure 4: Error type distribution of the base model on KoCo-Bench.
• Figure 5: Results of Performance-Cost Trade-off.