Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decoding Workload and Agreement From EEG During Spoken Dialogue With Conversational AI

Lucija Mihić Zidar, Philipp Wicke, Praneel Bhatia, Rosa Lutz, Marius Klug, Thorsten O. Zander

TL;DR

This study addresses implicit feedback for LLM alignment by decoding EEG signals of mental workload and implicit agreement during spoken dialogue with AI. It transfers classifiers trained in calibration tasks to two conversational paradigms—the Spelling Bee task for workload and a sentence-completion agreement task—while building a pipeline to synchronize word-level events with continuous EEG predictions. Results show workload decoding generalizes across domains, with one participant showing a significant linear increase across rounds ($p<0.001$, $R^2=0.79$). The agreement classifier produced continuous fluctuations during conversation rather than event-locked responses, highlighting boundary conditions for transferring event-driven pBCI signals to naturalistic dialogue. The work demonstrates feasibility and clarifies constraints for integrating passive EEG feedback into adaptive conversational AI, pointing to future improvements in event detection and multimodal cues.

Abstract

Passive brain-computer interfaces offer a potential source of implicit feedback for alignment of large language models, but most mental state decoding has been done in controlled tasks. This paper investigates whether established EEG classifiers for mental workload and implicit agreement can be transferred to spoken human-AI dialogue. We introduce two conversational paradigms - a Spelling Bee task and a sentence completion task- and an end-to-end pipeline for transcribing, annotating, and aligning word-level conversational events with continuous EEG classifier output. In a pilot study, workload decoding showed interpretable trends during spoken interaction, supporting cross-paradigm transfer. For implicit agreement, we demonstrate continuous application and precise temporal alignment to conversational events, while identifying limitations related to construct transfer and asynchronous application of event-based classifiers. Overall, the results establish feasibility and constraints for integrating passive BCI signals into conversational AI systems.

Decoding Workload and Agreement From EEG During Spoken Dialogue With Conversational AI

TL;DR

This study addresses implicit feedback for LLM alignment by decoding EEG signals of mental workload and implicit agreement during spoken dialogue with AI. It transfers classifiers trained in calibration tasks to two conversational paradigms—the Spelling Bee task for workload and a sentence-completion agreement task—while building a pipeline to synchronize word-level events with continuous EEG predictions. Results show workload decoding generalizes across domains, with one participant showing a significant linear increase across rounds (, ). The agreement classifier produced continuous fluctuations during conversation rather than event-locked responses, highlighting boundary conditions for transferring event-driven pBCI signals to naturalistic dialogue. The work demonstrates feasibility and clarifies constraints for integrating passive EEG feedback into adaptive conversational AI, pointing to future improvements in event detection and multimodal cues.

Abstract

Passive brain-computer interfaces offer a potential source of implicit feedback for alignment of large language models, but most mental state decoding has been done in controlled tasks. This paper investigates whether established EEG classifiers for mental workload and implicit agreement can be transferred to spoken human-AI dialogue. We introduce two conversational paradigms - a Spelling Bee task and a sentence completion task- and an end-to-end pipeline for transcribing, annotating, and aligning word-level conversational events with continuous EEG classifier output. In a pilot study, workload decoding showed interpretable trends during spoken interaction, supporting cross-paradigm transfer. For implicit agreement, we demonstrate continuous application and precise temporal alignment to conversational events, while identifying limitations related to construct transfer and asynchronous application of event-based classifiers. Overall, the results establish feasibility and constraints for integrating passive BCI signals into conversational AI systems.
Paper Structure (14 sections, 5 figures, 2 tables)

This paper contains 14 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Participants and procedure
  4. Calibration paradigms
  5. Conversational paradigms
  6. Workload Paradigm
  7. Agreement Paradigm
  8. Data acquisition
  9. Transcription and annotation
  10. EEG processing
  11. Calibration of workload and agreement predictive models
  12. Workload and agreement decoding in conversational paradigms
  13. Results
  14. Discussion

Figures (5)

  • Figure 1: Example scene stimulus (kitchen) used in the agreement paradigm.
  • Figure 2: Participant #1 - Round-level Workload in the Spelling Bee Paradigm (Means with AR(1)-aware 95% CI)
  • Figure 3: Participant #2 - Round-level Workload in the Spelling Bee Paradigm (Means with AR(1)-aware 95% CI)
  • Figure 4: Participant #3 - Agreement classifier output applied to its own training data from the grid task (raw predictive values smoothed with a moving average over 10 samples, decision boundary at 0). Correct and incorrect jump events marked with green and red, respectively.
  • Figure 5: Participant #3 - Agreement classifier output applied to the conversational paradigm (values normalized from -1 to 1, decision boundary at 0) with timestamps of word onset.