Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Alternative positional encoding functions for neural transformers

Ezequiel Lopez-Rubio, Macoris Decena-Gimenez, Rafael Marcos Luque-Baena

TL;DR

The paper introduces non-sinusoidal periodic functions (Triangular, Square, Sawtooth) as alternatives to the standard sinusoidal rotary positional embeddings (RoPE) in Transformer models. By enforcing period and phase-shift constraints, these encodings are integrated into RoPE-style rotations and evaluated on the Multi30K English–German translation task. Across 10-fold cross-validation, all three alternatives outperform the sinusoidal baseline in both loss and BLEU scores, with Sawtooth delivering the best BLEU and Triangular offering faster convergence. The findings suggest that alternative periodic encodings can improve performance and potentially reduce energy consumption, motivating broader validation across tasks and architectures.

Abstract

A key module in neural transformer-based deep architectures is positional encoding. This module enables a suitable way to encode positional information as input for transformer neural layers. This success has been rooted in the use of sinusoidal functions of various frequencies, in order to capture recurrent patterns of differing typical periods. In this work, an alternative set of periodic functions is proposed for positional encoding. These functions preserve some key properties of sinusoidal ones, while they depart from them in fundamental ways. Some tentative experiments are reported, where the original sinusoidal version is substantially outperformed. This strongly suggests that the alternative functions may have a wider use in other transformer architectures.

Alternative positional encoding functions for neural transformers

TL;DR

The paper introduces non-sinusoidal periodic functions (Triangular, Square, Sawtooth) as alternatives to the standard sinusoidal rotary positional embeddings (RoPE) in Transformer models. By enforcing period and phase-shift constraints, these encodings are integrated into RoPE-style rotations and evaluated on the Multi30K English–German translation task. Across 10-fold cross-validation, all three alternatives outperform the sinusoidal baseline in both loss and BLEU scores, with Sawtooth delivering the best BLEU and Triangular offering faster convergence. The findings suggest that alternative periodic encodings can improve performance and potentially reduce energy consumption, motivating broader validation across tasks and architectures.

Abstract

A key module in neural transformer-based deep architectures is positional encoding. This module enables a suitable way to encode positional information as input for transformer neural layers. This success has been rooted in the use of sinusoidal functions of various frequencies, in order to capture recurrent patterns of differing typical periods. In this work, an alternative set of periodic functions is proposed for positional encoding. These functions preserve some key properties of sinusoidal ones, while they depart from them in fundamental ways. Some tentative experiments are reported, where the original sinusoidal version is substantially outperformed. This strongly suggests that the alternative functions may have a wider use in other transformer architectures.
Paper Structure (5 sections, 8 equations, 3 figures, 1 table)

This paper contains 5 sections, 8 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experiments
  4. Discussion
  5. Conclusions

Figures (3)

  • Figure 1: The standard sinusoidal function and the proposed alternative functions.
  • Figure 2: Training dynamics of the loss across the four positional encoding variants. The plot shows the average training loss (solid lines) and validation loss (dashed lines) as a function of the training epoch. All curves are averaged over the 10 cross-validation folds.
  • Figure 3: Training dynamics of validation BLEU-4 across the four positional encoding variants. The plot shows the average validation BLEU-4 score as a function of the training epoch. All curves are averaged over the 10 cross-validation folds.