Structure-Preserving Transformers for Sequences of SPD Matrices

TL;DR

This work tackles learning from sequences of Symmetric Positive Definite matrices by preserving their Riemannian geometry throughout processing. It introduces SP-MHA, a structure-preserving multihead attention mechanism based on LogEuclidean mappings and triangular linear maps, integrated into SPDTransNet for EEG sleep staging. The approach preserves SPD structure across all Transformer components and achieves state-of-the-art macro-F1 and N1-F1 scores on the MASS SS3 dataset, outperforming several baselines and ablations confirm the benefit of structure preservation. The method offers a principled, geometry-aware framework for SPD-valued data with potential applicability beyond sleep staging to other domains requiring manifold-consistent sequence modeling.

Abstract

In recent years, Transformer-based auto-attention mechanisms have been successfully applied to the analysis of a variety of context-reliant data types, from texts to images and beyond, including data from non-Euclidean geometries. In this paper, we present such a mechanism, designed to classify sequences of Symmetric Positive Definite matrices while preserving their Riemannian geometry throughout the analysis. We apply our method to automatic sleep staging on timeseries of EEG-derived covariance matrices from a standard dataset, obtaining high levels of stage-wise performance.

Figures (2)

• Figure 1: The SP-MHA architecture. In parentheses are tensor dimensions at every step, with $N$ the batch size.
• Figure 2: SPDTransNet global architecture, with $t=3$ feature tokens per epoch.