Symbiotic Brain-Machine Drawing via Visual Brain-Computer Interfaces

TL;DR

This work shows that symbiotic human-AI interaction can significantly increase BCI bit-rates by more than a factor 5x, providing a platform for future development of AI-augmented BCI.

Abstract

Brain-computer interfaces (BCIs) are evolving from research prototypes into clinical, assistive, and performance enhancement technologies. Despite the rapid rise and promise of implantable technologies, there is a need for better and more capable wearable and non-invasive approaches whilst also minimising hardware requirements. We present a non-invasive BCI for mind-drawing that iteratively infers a subject's internal visual intent by adaptively presenting visual stimuli (probes) on a screen encoded at different flicker-frequencies and analyses the steady-state visual evoked potentials (SSVEPs). A Gabor-inspired or machine-learned policies dynamically update the spatial placement of the visual probes on the screen to explore the image space and reconstruct simple imagined shapes within approximately two minutes or less using just single-channel EEG data. Additionally, by leveraging stable diffusion models, reconstructed mental images can be transformed into realistic and detailed visual representations. Whilst we expect that similar results might be achievable with e.g. eye-tracking techniques, our work shows that symbiotic human-AI interaction can significantly increase BCI bit-rates by more than a factor 5x, providing a platform for future development of AI-augmented BCI.

Figures (7)

• Figure 1: Experimental setup and workflow. (a) Setup of the EEG-based mind-drawing system. The EEG device is custom-built and consists of a headband housing three wet electrodes (using saline solution to improve contact): two placed on the temples (ground and reference) and one at the occipital Oz position. The subject selects the disc that overlaps most with their imagined image. (b) Workflow of the mind-drawing process (Y: Yes; N: No). (c) Example screen-shot stimulus under the Gabor policy function, where discs are randomly arranged and each flickers at a unique frequency. (d) Example screen-shot of the stimulus under the data-driven policy function, where discs are randomly arranged, with some representing green features and others appearing as standalone elements.
• Figure 2: Imaging results from eight subjects, each targeting three imagined shapes. Panels are grouped three at a time, one group for each subject, i.e. panels (a–c) correspond to the first subject, (d–f) to the second subject, and so forth. Green-colored images represent the handwritten target after image resizing, while pink-colored images indicate the reconstructed images generated by our BCI system. Overlapped regions highlight the common areas between the two images, indicating reconstruction accuracy. We calculated the cosine similarity (COSS) between each ground-truth and reconstructed image pair to quantify accuracy. The overall average COSS across all subjects was 0.76 (± 0.04), indicating a good level of reconstruction fidelity.
• Figure 3: Distribution of image cosine similarity (COSS) values between target and reconstructed image pairs. (a) Box plot of COSS values for each subject. (b) Histogram of COSS values across all subjects.
• Figure 4: Mutual information (with standard deviation across iterations and subjects) increases with the number of iterations, although the rate of increase slows down as more iterations are added. The final MI reaches 222 bits starting from 91 bits (MI from an image with all black pixels), corresponding to an average rate of 1.31 bit/s.
• Figure 5: Reconstruction results for the MNIST digit “7.” The first row (a–d) shows the step-by-step reconstruction of the digit from SSVEP signals by placing a Gaussian disc at the location of each disc selected by the SSVEP signal. The second row (f–i) presents the corresponding images generated by the system at each step. For simplicity, we only show the images every 2 iteration steps. Panels (e) and (j) overlay the binarised reconstructed and predicted images with the aligned handwritten target in different colour channels: green indicates the aligned handwritten target, pink represents the reconstructed mental image, and overlapped regions highlight areas of agreement, demonstrating reconstruction accuracy. Grey regions indicate agreement, while colored regions highlight intensity differences. The COSS values for the binarised reconstructed and predicted images are 0.75 and 0.78, respectively , indicating a high degree of similarity to the target image.