Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EEG-Based Brain-LLM Interface for Human Preference Aligned Generation

Junzi Zhang, Jianing Shen, Weijie Tu, Yi Zhang, Hailin Zhang, Tom Gedeon, Bin Jiang, Yue Yao

Abstract

Large language models (LLMs) are becoming an increasingly important component of human--computer interaction, enabling users to coordinate a wide range of intelligent agents through natural language. While language-based interfaces are powerful and flexible, they implicitly assume that users can reliably produce explicit linguistic input, an assumption that may not hold for users with speech or motor impairments, e.g., Amyotrophic Lateral Sclerosis (ALS). In this work, we investigate whether neural signals can be used as an alternative input to LLMs, particularly to support those socially marginalized or underserved users. We build a simple brain-LLM interface, which uses EEG signals to guide image generation models at test time. Specifically, we first train a classifier to estimate user satisfaction from EEG signals. Its predictions are then incorporated into a test-time scaling (TTS) framework that dynamically adapts model inference using neural feedback collected during user evaluation. The experiments show that EEG can predict user satisfaction, suggesting that neural activity carries information on real-time preference inference. These findings provide a first step toward integrating neural feedback into adaptive language-model inference, and hopefully open up new possibilities for future research on adaptive LLM interaction.

EEG-Based Brain-LLM Interface for Human Preference Aligned Generation

Abstract

Large language models (LLMs) are becoming an increasingly important component of human--computer interaction, enabling users to coordinate a wide range of intelligent agents through natural language. While language-based interfaces are powerful and flexible, they implicitly assume that users can reliably produce explicit linguistic input, an assumption that may not hold for users with speech or motor impairments, e.g., Amyotrophic Lateral Sclerosis (ALS). In this work, we investigate whether neural signals can be used as an alternative input to LLMs, particularly to support those socially marginalized or underserved users. We build a simple brain-LLM interface, which uses EEG signals to guide image generation models at test time. Specifically, we first train a classifier to estimate user satisfaction from EEG signals. Its predictions are then incorporated into a test-time scaling (TTS) framework that dynamically adapts model inference using neural feedback collected during user evaluation. The experiments show that EEG can predict user satisfaction, suggesting that neural activity carries information on real-time preference inference. These findings provide a first step toward integrating neural feedback into adaptive language-model inference, and hopefully open up new possibilities for future research on adaptive LLM interaction.
Paper Structure (17 sections, 3 equations, 8 figures, 2 tables, 1 algorithm)

This paper contains 17 sections, 3 equations, 8 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Brain-LLM Interface Dataset
  4. Participants
  5. Stimuli Generation
  6. EEG Data Collection
  7. Method
  8. Recap of Test-Time Scaling
  9. The Brain-LLM Interface for TTS
  10. Experiment
  11. EEG-based Satisfaction Classification
  12. Temporal dynamics confirm cognitive origin.
  13. Analysis of Brain Regions
  14. Model Alignment with Human Preference
  15. Brain-LLM consistently improves alignment and generation quality over baseline.
  16. ...and 2 more sections

Figures (8)

  • Figure 1: Motivation for EEG-based interface for LLM. Existing large language models rely mainly on explicit linguistic feedback such as typed instructions or corrections. However, there may exist a group of users with speech or motor impairments (e.g., ALS) thus language may not be available. For those users, we investigate whether neural signals can be used as an alternative input to LLMs.
  • Figure 2: Meta-information on the semantic themes, challenge categories, and EEG data collection procedure of BLID.(A) Visualization of the six semantic themes and fourteen challenging aspects of text-to-image generation used to construct the stimulus set. (B) Balanced distribution of prompt–image pairs across six semantic themes and fourteen challenge categories, ensuring diverse coverage of both objective constraints and subjective nuances. (C) Illustration of the EEG collection procedure. The experiment consisted of five evaluation blocks ($12$ minutes each). Each block contained $26$ trials, and each trial lasted approximately $28$ seconds, including stimulus presentation, evaluation periods, and inter-trial intervals.
  • Figure 3: Overview of the Brain-LLM interface.Left: Participants view image stimuli while wearing a 64-channel EEG cap; Right: the signals are recorded and amplified to train an EEG foundation model that predicts user satisfaction. During inference, a generation model produces an initial answer, and the EEG model monitors the user’s neural response, classifying each trial as "Satisfied" or "Not satisfied". If not satisfied, the system triggers test time scaling step and loops until the user is satisfied or a maximum number of iterations is reached.
  • Figure 4: Time-resolved decoding analysis of user satisfaction. We evaluate the decoding accuracy of representative hand-crafted (SVM) and end-to-end (LaBraM) models using a sliding window approach relative to the button press (0.0s). The shaded regions represent the standard deviation across participants.
  • Figure 5: Topographical maps of single-channel classification accuracy for four participants. Warm colors indicate electrodes with higher decoding accuracy, and cool colors indicate lower accuracy. While individual variability is present, several regions show consistently higher performance across participants.
  • ...and 3 more figures