NeuroSketch: An Effective Framework for Neural Decoding via Systematic Architectural Optimization

TL;DR

NeuroSketch addresses the gap in neural-decoding architecture exploration by systematically optimizing a CNN-2D–based framework from macro to micro levels. Through extensive experiments across EEG/SEEG/ECoG modalities and eight decoding tasks, it demonstrates state-of-the-art performance and scalable gains, particularly on challenging datasets like Du-IN. The framework combines a patch-aware CNN-2D base, stepwise feature-map growth, pagoda downsampling, and group convolutions, yielding efficient yet powerful representations validated by neurophysiological interpretability analyses. These findings provide a practical, scalable approach for neural decoding and offer insights into architecture-design principles tailored to rapid neural dynamics and localized brain activations.

Abstract

Neural decoding, a critical component of Brain-Computer Interface (BCI), has recently attracted increasing research interest. Previous research has focused on leveraging signal processing and deep learning methods to enhance neural decoding performance. However, the in-depth exploration of model architectures remains underexplored, despite its proven effectiveness in other tasks such as energy forecasting and image classification. In this study, we propose NeuroSketch, an effective framework for neural decoding via systematic architecture optimization. Starting with the basic architecture study, we find that CNN-2D outperforms other architectures in neural decoding tasks and explore its effectiveness from temporal and spatial perspectives. Building on this, we optimize the architecture from macro- to micro-level, achieving improvements in performance at each step. The exploration process and model validations take over 5,000 experiments spanning three distinct modalities (visual, auditory, and speech), three types of brain signals (EEG, SEEG, and ECoG), and eight diverse decoding tasks. Experimental results indicate that NeuroSketch achieves state-of-the-art (SOTA) performance across all evaluated datasets, positioning it as a powerful tool for neural decoding. Our code and scripts are available at https://github.com/Galaxy-Dawn/NeuroSketch.

Figures (6)

• Figure 1: Roadmap of architectural optimization.
• Figure 2: (a) Comparison of different patch methods. Vanilla GRU and Transformer treat a single timestamp across all channels as a token; PatchTST treats multiple timestamps of a single channel as a token; iTransformer treats all timestamps of an entire channel as a token. We propose a simple but effective patch method that aggregates information of multiple timestamps and channels.(b) Comparison of different latent space transformation methods. Regarding the number of feature maps, the step approach increases them gradually, whereas the leap approach increases them rapidly in the early stages. In terms of their size, the pyramid approach decreases them progressively, while the pagoda approach decreases them quickly in the early stages. (c) The overall architecture of NeuroSketch.
• Figure 3: Model performance comparison.
• Figure 4: Score‑CAM visualization of all channels for Subjects 02 and 11 in the Du‑IN dataset generated by NeuroSketch-Large.
• Figure 5: Layer‑wise $\varepsilon$‑Rank Ratio Analysis of MedFormer and NeuroSketch-Large.