Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HAFFormer: A Hierarchical Attention-Free Framework for Alzheimer's Disease Detection From Spontaneous Speech

Zhongren Dong, Zixing Zhang, Weixiang Xu, Jing Han, Jianjun Ou, Björn W. Schuller

TL;DR

The paper tackles the challenge of detecting Alzheimer's Disease (AD) from long, spontaneous speech by moving away from quadratic self-attention. It introduces HAFFormer, a Hierarchical Attention-Free Transformer that uses a multi-scale depthwise convolution Token Mixer and a GELU-based gated linear Channel Mixer, arranged in a hierarchy of merges to capture information from frame to dialogue level. On the ADReSS-M dataset, HAFFormer achieves competitive accuracy (about 82.6%) with a substantially smaller model size and lower computational cost than traditional Transformer approaches, validating its efficiency for edge deployment on long audio. The approach leverages cross-lingual SSL features (Wav2Vec2 XLS-R and WhisperX segmentation) to handle multilingual spontaneous speech, offering a practical, scalable solution for early AD screening in mobile or low-resource settings.

Abstract

Automatically detecting Alzheimer's Disease (AD) from spontaneous speech plays an important role in its early diagnosis. Recent approaches highly rely on the Transformer architectures due to its efficiency in modelling long-range context dependencies. However, the quadratic increase in computational complexity associated with self-attention and the length of audio poses a challenge when deploying such models on edge devices. In this context, we construct a novel framework, namely Hierarchical Attention-Free Transformer (HAFFormer), to better deal with long speech for AD detection. Specifically, we employ an attention-free module of Multi-Scale Depthwise Convolution to replace the self-attention and thus avoid the expensive computation, and a GELU-based Gated Linear Unit to replace the feedforward layer, aiming to automatically filter out the redundant information. Moreover, we design a hierarchical structure to force it to learn a variety of information grains, from the frame level to the dialogue level. By conducting extensive experiments on the ADReSS-M dataset, the introduced HAFFormer can achieve competitive results (82.6% accuracy) with other recent work, but with significant computational complexity and model size reduction compared to the standard Transformer. This shows the efficiency of HAFFormer in dealing with long audio for AD detection.

HAFFormer: A Hierarchical Attention-Free Framework for Alzheimer's Disease Detection From Spontaneous Speech

TL;DR

The paper tackles the challenge of detecting Alzheimer's Disease (AD) from long, spontaneous speech by moving away from quadratic self-attention. It introduces HAFFormer, a Hierarchical Attention-Free Transformer that uses a multi-scale depthwise convolution Token Mixer and a GELU-based gated linear Channel Mixer, arranged in a hierarchy of merges to capture information from frame to dialogue level. On the ADReSS-M dataset, HAFFormer achieves competitive accuracy (about 82.6%) with a substantially smaller model size and lower computational cost than traditional Transformer approaches, validating its efficiency for edge deployment on long audio. The approach leverages cross-lingual SSL features (Wav2Vec2 XLS-R and WhisperX segmentation) to handle multilingual spontaneous speech, offering a practical, scalable solution for early AD screening in mobile or low-resource settings.

Abstract

Automatically detecting Alzheimer's Disease (AD) from spontaneous speech plays an important role in its early diagnosis. Recent approaches highly rely on the Transformer architectures due to its efficiency in modelling long-range context dependencies. However, the quadratic increase in computational complexity associated with self-attention and the length of audio poses a challenge when deploying such models on edge devices. In this context, we construct a novel framework, namely Hierarchical Attention-Free Transformer (HAFFormer), to better deal with long speech for AD detection. Specifically, we employ an attention-free module of Multi-Scale Depthwise Convolution to replace the self-attention and thus avoid the expensive computation, and a GELU-based Gated Linear Unit to replace the feedforward layer, aiming to automatically filter out the redundant information. Moreover, we design a hierarchical structure to force it to learn a variety of information grains, from the frame level to the dialogue level. By conducting extensive experiments on the ADReSS-M dataset, the introduced HAFFormer can achieve competitive results (82.6% accuracy) with other recent work, but with significant computational complexity and model size reduction compared to the standard Transformer. This shows the efficiency of HAFFormer in dealing with long audio for AD detection.
Paper Structure (15 sections, 2 equations, 4 figures, 2 tables)

This paper contains 15 sections, 2 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Hierarchical Attention-Free Transformer
  3. Speech Preprocessing and Embedding
  4. Projection
  5. Merge
  6. AFFormer block
  7. Experiments and Results
  8. Dataset
  9. Experimental Setup
  10. Results and Discussions
  11. Comparison with SOTA Approaches
  12. Selection of the Token Mixers
  13. Selection of the Channel Mixers
  14. Selection of the Number of Hierarchy
  15. CONCLUSION

Figures (4)

  • Figure 1: An overview of the proposed Hierarchical Attention-Free Transformer (HAFFormer) framework for Alzheimer's disease detection. MSDW: multi-scale depth-wise convolution; GEGLU: GELU-based Gated Linear Units; AD/HC: Alzheimer's Disease or Healthy Control.
  • Figure 2: Detailed architecture of the MSDW module (a) and the GEGLU module (b).
  • Figure 3: Model performance (accuracy) vs model size (number of parameters) when taking different types of token mixers under the same channel mixer (i. e., FFN). MACs : Multiply–accumulate operations.
  • Figure 4: Performance of the proposed HAFFormer when taking different numbers of hierarchy. 3-1/2 indicates the three hierarchies but the last one contains one or two blocks.