Do Code LLMs Do Static Analysis?
Chia-Yi Su, Collin McMillan
TL;DR
This paper investigates whether code LLMs perform static analysis, a core human programming activity, by evaluating call-graph, dataflow, and AST generation alongside code summarization, translation, and generation. It compares open-source (JAM, CodeLLaMA) and closed-source (GPT-4o, Gemini) models across Java and C/C++, employing finetuning and in-context learning to probe static-analysis capabilities. The main findings show that code LLMs are generally poor at static analysis and that pretraining on static-analysis tasks does not reliably improve code tasks, though finetuning can boost simple static-analysis tasks like AST generation. These results challenge assumptions about human-like reasoning in LLMs for software engineering and suggest that future model design should rethink how static-analysis capabilities are trained and integrated into code intelligence workflows.
Abstract
This paper investigates code LLMs' capability of static analysis during code intelligence tasks such as code summarization and generation. Code LLMs are now household names for their abilities to do some programming tasks that have heretofore required people. The process that people follow to do programming tasks has long been understood to require static analysis. For example, human programmers navigate the call graph of large programs to comprehend the different parts of those programs. Education in programming includes static analysis under the assumption that better static analysis skills beget better programming. Yet while popular culture is replete with anthropomorphic references such as LLM "reasoning", in fact code LLMs could exhibit a wholly alien thought process to humans. This paper studies the specific question of static analysis by code LLMs. We use three different static analysis tasks (callgraph generation, AST generation, and dataflow generation) and three different code intelligence tasks (code generation, summarization, and translation) with two different open-source models (Gemini and GPT-4o) and closed-source models (CodeLlaMA and Jam) as our experiments. We found that LLMs show poor performance on static analysis tasks and that pretraining on the static analysis tasks does not generalize to better performance on the code intelligence tasks.