CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks

TL;DR

CoRe delivers a multilingual static-analysis benchmark to probe LLMs' code-reasoning capabilities over data dependency, control dependency, and information flow. It employs semantics-aware sampling to generate 12,553 task instances from 180 annotated programs in C/C++, Java, and Python, with a semi-automatic annotation pipeline to ensure high-quality ground truth. Evaluating 10 models, the study finds that while dependency identification is generally strong for reasoning models, deeper semantic tasks like trace generation and information-flow reasoning remain bottlenecks, especially for longer and more complex code. The benchmark offers a fine-grained, language-rich framework to drive improvements in LLM-based code reasoning and suggests directions such as inter-procedural analysis and robustness against complex control structures and adversarial prompts.

Abstract

Large language models (LLMs) have been widely adopted across diverse domains of software engineering, such as code generation, program repair, and vulnerability detection. These applications require understanding beyond surface-level code patterns: value propagation, control flow, and interdependence between program elements. However, existing benchmarks primarily evaluate end-to-end outcomes, such as whether code is correctly repaired or generated, leaving the models' ability for program semantic reasoning underexplored. This work presents CORE, a high-quality, human-verified benchmark designed to evaluate LLMs on fundamental static analysis tasks. CORE includes 12,553 task instances spanning data dependency, control dependency, and information flow across programs written in C/C++, Java, and Python. To ensure semantic diversity and reasoning complexity, we propose a semantics-aware diverse sampling strategy that selects targets and task instances based on structural coverage and dependency depth. We evaluate 10 mainstream LLMs and show that, while they perform well at identifying dependencies, models still struggle with tasks that require deeper semantic understanding and multi-step reasoning. We further conduct qualitative analyses to uncover key challenges, such as complex control structures and backward dependency patterns, offering insights into improving LLMs' code reasoning capabilities.

Figures (5)

• Figure 1: Real-world motivating examples for data dependency, control dependency, and information flow in security and software engineering applications.
• Figure 2: Performance distribution of reasoning vs. non-reasoning models across all three task types: Data Dependency (Data), Control Dependency (Control), and Information Flow (InfoFlow).
• Figure 3: Factors affect model performance, with longer functions, complex control, and reverse dependencies leading to consistent drops.
• Figure 4: Impact of query type (classification only vs. with trace) and input clipping on F1 score and correct trace rate across tasks.
• Figure 5: F1 and correct trace rate under FSL as the number of retrieved examples ($k$) increases.