NoiseFormer -- Noise Diffused Symmetric Attention Transformer

TL;DR

NoiseFormer introduces Noise Diffused Symmetric Attention as a unified framework that treats generic attention as a noisy version of symmetric attention, enabling memory-efficient transformer blocks with minimal overhead. By jointly learning additive noise statistics and an optimal estimator for generic attention scores within the same model, it achieves competitive performance while preserving the memory advantages of symmetric attention. The approach is validated on a GPT-2–based NanoGPT, showing parameter reductions around 5.8% and competitive GLUE performance compared to both the base model and Symmetric Attention baselines, with several variants exploring shared versus per-head noise. The work suggests practical benefits for scalable autoregressive models and opens avenues for diversified inference and input-conditioned noise. Overall, NoiseFormer offers a principled, low-overhead path to bridge the gap between expressive attention and efficient deployment in large-scale transformers.

Abstract

Transformer architecture has been very successful long runner in the field of Deep Learning (DL) and Large Language Models (LLM) because of its powerful attention-based learning and parallel-natured architecture. As the models grow gigantic in terms of memory footprint, difficulties in fitting the model on a device like a GPU or an AI accelerator give rise to the need for multiple computing devices thereby escalating the computing cost. This increased training/inference cost paved the way for efficient model size reduction/parametric reduction deploying Sparse Attention techniques. In this paper, we start analyzing one of the techniques of Sparse Attention called Symmetric Dot-Product Attention (referred to as Symmetric Attention) and propose a novel unified model architecture called Noise Diffused Symmetric Attention Transformer to enhance the model's performance. While maintaining the memory gains of Symmetric Attention, with minute overhead in terms of model parameters and computational overhead, the proposed model brings in enhanced performance in terms of accuracy and inference-time sampling. The proposed model is validated upon GPT2 base model and the results reflect the performance gains falling between plain Symmetric attention and GPT2 base model on a variety of GLUE benchmark tasks in terms of accuracy, with significant model size reduction with respect to the base model.

Figures (10)

• Figure 1: Conventional Attention in Transformers sym_attention
• Figure 2: Conventional Multi-Head Attention ($MHA$) in Transformers
• Figure 3: Symmetric Multi-Head Attention sym_attention
• Figure 4: Single head Attention in Proposed NoiseFormer
• Figure 5: All-heads Shared Noisy Symmetric Attention (All-heads SNSA) in a NoiseFormer layer