Towards Attention-Aware Large Language Models: Integrating Real-Time Eye-Tracking and EEG for Adaptive AI Responses

TL;DR

The paper addresses static LLM interfaces that ignore fluctuating user attention and proposes a real-time neuroadaptive solution by fusing EEG and eye-tracking Signals to classify five attentional states. It presents an end-to-end pipeline for data collection, preprocessing, feature extraction, and multimodal fusion into an attention classifier that drives LLM prompt adaptation across states. A within-subject user study compares adaptive versus baseline LLMs across decision making, creative problem solving, debating, and information seeking tasks, using NASA-TLX and qualitative interviews to assess engagement and cognitive load. The work aims for at least 70% state-classification accuracy and seeks to demonstrate practical benefits of neuroadaptive LLMs for improved user experience and reduced cognitive burden, signaling a feasible path toward real-time attention-aware AI assistants.

Abstract

This project proposes an attention-aware LLM that integrates EEG and eye tracking to monitor and measure user attention dynamically. To realize this, the project will integrate real-time EEG and eye-tracking data into an LLM-based interactive system and classify the user's attention state on the fly. The system can identify five attention states: High Attention, Stable Attention, Dropping Attention, Cognitive Overload, and Distraction. It responds accordingly to each state, with a particular focus on adapting to decreased attention, distraction, and cognitive overload to improve user engagement and reduce cognitive load.