Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EyeLayer: Integrating Human Attention Patterns into LLM-Based Code Summarization

Jiahao Zhang, Yifan Zhang, Kevin Leach, Yu Huang

TL;DR

EyeLayer is proposed, a lightweight attention-augmentation module that incorporates human eye-gaze patterns, as a proxy of human expertise, into LLM-based code summarization, demonstrating that human gaze patterns encode complementary attention signals that enhance the semantic focus of LLMs and transfer effectively across diverse models for code summarization.

Abstract

Code summarization is the task of generating natural language descriptions of source code, which is critical for software comprehension and maintenance. While large language models (LLMs) have achieved remarkable progress on this task, an open question remains: can human expertise in code understanding further guide and enhance these models? We propose EyeLayer, a lightweight attention-augmentation module that incorporates human eye-gaze patterns, as a proxy of human expertise, into LLM-based code summarization. EyeLayer models human attention during code reading via a Multimodal Gaussian Mixture, redistributing token embeddings based on learned parameters (μ_i, σ_i^2) that capture where and how intensively developers focus. This design enables learning generalizable attention priors from eye-tracking data and incorporating them into LLMs seamlessly, without disturbing existing representations. We evaluate EyeLayer across diverse model families (i.e., LLaMA-3.2, Qwen3, and CodeBERT) covering different scales and architectures. EyeLayer consistently outperforms strong fine-tuning baselines across standard metrics, achieving gains of up to 13.17% on BLEU-4. These results demonstrate that human gaze patterns encode complementary attention signals that enhance the semantic focus of LLMs and transfer effectively across diverse models for code summarization.

EyeLayer: Integrating Human Attention Patterns into LLM-Based Code Summarization

TL;DR

EyeLayer is proposed, a lightweight attention-augmentation module that incorporates human eye-gaze patterns, as a proxy of human expertise, into LLM-based code summarization, demonstrating that human gaze patterns encode complementary attention signals that enhance the semantic focus of LLMs and transfer effectively across diverse models for code summarization.

Abstract

Code summarization is the task of generating natural language descriptions of source code, which is critical for software comprehension and maintenance. While large language models (LLMs) have achieved remarkable progress on this task, an open question remains: can human expertise in code understanding further guide and enhance these models? We propose EyeLayer, a lightweight attention-augmentation module that incorporates human eye-gaze patterns, as a proxy of human expertise, into LLM-based code summarization. EyeLayer models human attention during code reading via a Multimodal Gaussian Mixture, redistributing token embeddings based on learned parameters (μ_i, σ_i^2) that capture where and how intensively developers focus. This design enables learning generalizable attention priors from eye-tracking data and incorporating them into LLMs seamlessly, without disturbing existing representations. We evaluate EyeLayer across diverse model families (i.e., LLaMA-3.2, Qwen3, and CodeBERT) covering different scales and architectures. EyeLayer consistently outperforms strong fine-tuning baselines across standard metrics, achieving gains of up to 13.17% on BLEU-4. These results demonstrate that human gaze patterns encode complementary attention signals that enhance the semantic focus of LLMs and transfer effectively across diverse models for code summarization.
Paper Structure (36 sections, 11 equations, 5 figures, 3 tables)

This paper contains 36 sections, 11 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Eye-tracking for Program Comprehension
  4. Probabilistic Attention and Cognitive Priors
  5. Methodology
  6. Datasets and Preprocessing
  7. Multimodal Gaussian EyeLayer
  8. Code-Level Embedding
  9. Sparse Gating Mechanism
  10. Mode-Specific Parameterization
  11. Gaussian Mixture Construction
  12. Causal-Aware Attention Redistribution
  13. Low-Rank Transformation
  14. Attention-Guided Weighting
  15. Adaptive Gating and Integration
  16. ...and 21 more sections

Figures (5)

  • Figure 1: Overview of our joint training pipeline.
  • Figure 2: The Multimodal Gaussian EyeLayer architecture.
  • Figure 3: Integration of the EyeLayer into transformer architectures for code summarization. Note that since CodeBERT is an encoder-only model, an auxiliary decoder is attached for sequence generation in the code summarization task.
  • Figure 4: Example from CodeXGLUE illustrating EyeLayer’s improvement over the baseline. Depict the inferred gaze-inspired attention across semantically related code regions.
  • Figure 5: Performance of Llama3.2-1B-Instruct when the EyeLayer is inserted at different transformer layers.