Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fusion Matters: Length-Aware Analysis of Positional-Encoding Fusion in Transformers

Mohamed Amine Hallam, Kuo-Kun Tseng

TL;DR

The results indicate that positional-encoding fusion is a non-trivial design choice for long-sequence Transformers and should be treated as an explicit modeling decision rather than a fixed default.

Abstract

Transformers require positional encodings to represent sequence order, yet most prior work focuses on designing new positional encodings rather than examining how positional information is fused with token embeddings. In this paper, we study whether the fusion mechanism itself affects performance, particularly in long-sequence settings. We conduct a controlled empirical study comparing three canonical fusion strategies--element-wise addition, concatenation with projection, and scalar gated fusion--under identical Transformer architectures, data splits, and random seeds. Experiments on three text classification datasets spanning short (AG News), medium (IMDB), and long (ArXiv) sequences show that fusion choice has negligible impact on short texts but produces consistent gains on long documents. To verify that these gains are structural rather than stochastic, we perform paired-seed analysis and cross-dataset comparison across sequence-length regimes. Additional experiments on the ArXiv dataset indicate that the benefit of learnable fusion generalizes across multiple positional encoding families. Finally, we explore a lightweight convolutional gating mechanism that introduces local inductive bias at the fusion level, evaluated on long documents only. Our results indicate that positional-encoding fusion is a non-trivial design choice for long-sequence Transformers and should be treated as an explicit modeling decision rather than a fixed default.

Fusion Matters: Length-Aware Analysis of Positional-Encoding Fusion in Transformers

TL;DR

The results indicate that positional-encoding fusion is a non-trivial design choice for long-sequence Transformers and should be treated as an explicit modeling decision rather than a fixed default.

Abstract

Transformers require positional encodings to represent sequence order, yet most prior work focuses on designing new positional encodings rather than examining how positional information is fused with token embeddings. In this paper, we study whether the fusion mechanism itself affects performance, particularly in long-sequence settings. We conduct a controlled empirical study comparing three canonical fusion strategies--element-wise addition, concatenation with projection, and scalar gated fusion--under identical Transformer architectures, data splits, and random seeds. Experiments on three text classification datasets spanning short (AG News), medium (IMDB), and long (ArXiv) sequences show that fusion choice has negligible impact on short texts but produces consistent gains on long documents. To verify that these gains are structural rather than stochastic, we perform paired-seed analysis and cross-dataset comparison across sequence-length regimes. Additional experiments on the ArXiv dataset indicate that the benefit of learnable fusion generalizes across multiple positional encoding families. Finally, we explore a lightweight convolutional gating mechanism that introduces local inductive bias at the fusion level, evaluated on long documents only. Our results indicate that positional-encoding fusion is a non-trivial design choice for long-sequence Transformers and should be treated as an explicit modeling decision rather than a fixed default.
Paper Structure (25 sections, 6 equations, 7 figures, 3 tables)

This paper contains 25 sections, 6 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Base Model and Positional Encoding
  5. Positional-Encoding Fusion Operators
  6. Additive fusion (Add)
  7. Concatenation with projection (Concat).
  8. Gated Addition (Gate-Scalar).
  9. Local Convolutional Positional Gating
  10. Experimental Setup
  11. Dataset
  12. Protocol
  13. Fusion Benchmark Lengths
  14. Robustness Across Sequence-Length Regimes
  15. Paired-Seed Delta Analysis
  16. ...and 10 more sections

Figures (7)

  • Figure 1: Fusion operators in the evaluated experiments.
  • Figure 2: Local convolutional positional gating (Gate-CNN). The gate is computed from a depth-wise 1D convolution over positional encodings only, then used to mix token and positional features.
  • Figure 3: Paired per-seed accuracy deltas (Gate Scalar $-$ Add) on the ArXiv dataset.
  • Figure 4: Mean accuracy and inference latency across fusion mechanisms.
  • Figure 5: Paired per-seed accuracy deltas on ArXiv relative to Add. Positive values indicate improvement over Add under the same seed.
  • ...and 2 more figures