Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?
Bonan Kou, Shengmai Chen, Zhijie Wang, Lei Ma, Tianyi Zhang
TL;DR
This study interrogates whether six large language models attending to task descriptions mirror human programmers when generating code. By constructing a programmer-attention dataset on 1,138 Python tasks and evaluating twelve attention-calculation methods across perturbation, gradient, and self-attention approaches, the authors reveal a robust misalignment between model and human attention, with only a minority of errors attributable to attentional misalignment. A perturbation-based method (BERT-masking) provides the best alignment and a user study shows programmers prefer such explanations, though trust remains limited. The work yields practical guidance for choosing attention-calculation techniques, suggests directions for human-aligned LLMs and attention-aware training, and provides a public dataset to foster interpretability in code generation.
Abstract
Large Language Models (LLMs) have recently been widely used for code generation. Due to the complexity and opacity of LLMs, little is known about how these models generate code. We made the first attempt to bridge this knowledge gap by investigating whether LLMs attend to the same parts of a task description as human programmers during code generation. An analysis of six LLMs, including GPT-4, on two popular code generation benchmarks revealed a consistent misalignment between LLMs' and programmers' attention. We manually analyzed 211 incorrect code snippets and found five attention patterns that can be used to explain many code generation errors. Finally, a user study showed that model attention computed by a perturbation-based method is often favored by human programmers. Our findings highlight the need for human-aligned LLMs for better interpretability and programmer trust.