CodeSimpleQA: Scaling Factuality in Code Large Language Models

TL;DR

<3-5 sentence high-level summary> CodeSimpleQA addresses the lack of evaluative focus on factual knowledge in code-oriented LLMs by introducing a bilingual, code-specific QA benchmark grounded in official documentation. It pairs CodeSimpleQA with a large CodeSimpleQA-Instruct post-training corpus and a post-training framework that combines supervised fine-tuning and reinforcement learning (GRPO) to improve factual code knowledge. Across English and Chinese domains, the study shows frontier LLMs still struggle with code factuality, but the proposed RFT/RL framework yields substantial improvements and highlights the complementary roles of retrieval-augmented approaches and instruction tuning. The work emphasizes factuality-aware alignment as essential for reliable, production-ready code AI systems and provides a scalable approach to advancing this capability.

Abstract

Large language models (LLMs) have made significant strides in code generation, achieving impressive capabilities in synthesizing code snippets from natural language instructions. However, a critical challenge remains in ensuring LLMs generate factually accurate responses about programming concepts, technical implementations, etc. Most previous code-related benchmarks focus on code execution correctness, overlooking the factual accuracy of programming knowledge. To address this gap, we present CodeSimpleQA, a comprehensive bilingual benchmark designed to evaluate the factual accuracy of code LLMs in answering code-related questions, which contains carefully curated question-answer pairs in both English and Chinese, covering diverse programming languages and major computer science domains. Further, we create CodeSimpleQA-Instruct, a large-scale instruction corpus with 66M samples, and develop a post-training framework combining supervised fine-tuning and reinforcement learning. Our comprehensive evaluation of diverse LLMs reveals that even frontier LLMs struggle with code factuality. Our proposed framework demonstrates substantial improvements over the base model, underscoring the critical importance of factuality-aware alignment in developing reliable code LLMs.

Figures (10)

• Figure 1: Comparison between the code generation task and the code factual question answering task.
• Figure 2: Data statistics for CodeSimpleQA.
• Figure 3: Examples of different domains in CodeSimpleQA.
• Figure 4: The framework to create the in CodeSimpleQA.
• Figure 5: Domains statistics in CodeSimpleQA.