EyeTrans: Merging Human and Machine Attention for Neural Code Summarization

TL;DR

EyeTrans tackles the challenge of improving neural code summarization by incorporating human attention gathered from eye-tracking into Transformer self-attention. It introduces a data-centric embedding approach that maps attention-switch edges over ASTs into token embeddings, enabling the Transformer to align machine attention with human cognition without altering the core architecture. The study reports substantial gains in functional and general code summarization, robustness to training noise, and interpretable changes in attention maps, while providing open eye-tracking data and code for replication. This work highlights a promising direction for human-centered AI in software engineering, suggesting that human gaze patterns can guide and simplify neural attention during training, with potential applicability to larger Transformer-based systems.

Abstract

Neural code summarization leverages deep learning models to automatically generate brief natural language summaries of code snippets. The development of Transformer models has led to extensive use of attention during model design. While existing work has primarily and almost exclusively focused on static properties of source code and related structural representations like the Abstract Syntax Tree (AST), few studies have considered human attention, that is, where programmers focus while examining and comprehending code. In this paper, we develop a method for incorporating human attention into machine attention to enhance neural code summarization. To facilitate this incorporation and vindicate this hypothesis, we introduce EyeTrans, which consists of three steps: (1) we conduct an extensive eye-tracking human study to collect and pre-analyze data for model training, (2) we devise a data-centric approach to integrate human attention with machine attention in the Transformer architecture, and (3) we conduct comprehensive experiments on two code summarization tasks to demonstrate the effectiveness of incorporating human attention into Transformers. Integrating human attention leads to an improvement of up to 29.91% in Functional Summarization and up to 6.39% in General Code Summarization performance, demonstrating the substantial benefits of this combination. We further explore performance in terms of robustness and efficiency by creating challenging summarization scenarios in which EyeTrans exhibits interesting properties. We also visualize the attention map to depict the simplifying effect of machine attention in the Transformer by incorporating human attention. This work has the potential to propel AI research in software engineering by introducing more human-centered approaches and data.

Figures (8)

• Figure 1: Conceptual Overview of EyeTrans. EyeTrans synthesizes human attention information into the self-attention mechanism of Transformers, assisting Transformers in code summarization tasks.
• Figure 2: (Left) The experimental room, with the task displayed on the monitor. The Tobii Pro Fusion Eye-tracker is a thin bar magnetized to a strip at the bottom of the monitor. (Right) A screenshot of one task example, with the Java method displayed on the left of the screen, and the summary writing location in the top right.
• Figure 3: EyeTrans Methodology Overview. (a) depicts our data collection processes. (b) describes the data preprocessing steps, including transformation to ASTs, data augmentation, and representing attention switches as edges on ASTs. (c) underlines our proposed approach to model attention switches using Transformers' default input embedding modality.
• Figure 4: Illustration of how EyeTrans models attention switch. Attention switches, conceptualized as edges connecting AST nodes, are mapped onto corresponding AST tokens by merging their respective embeddings.
• Figure 5: Count and Parent Token Length Distribution by Rating Source. The count decreases for all rating sources as filtering tightens. The consensus approach mitigates outliers, which are seen in strictly filtered comments from Rater A.