RingFormer: Rethinking Recurrent Transformer with Adaptive Level Signals

TL;DR

RingFormer addresses the high parameter cost of Transformers by reusing a single Transformer block in a ring-like recurrent fashion and introducing input-dependent, depth-aware level signals generated via low-rank transformations. The signals modulate the attention and feedforward projections ($Q_i= W_Q x + g_{A_{Qi}}(x)$, $K_i= W_K x + g_{A_{Ki}}(x)$, $V_i= W_V x + g_{A_{Vi}}(x)$, and $FFN(x)=\sigma((x+g_{Fi}(x))W_{up})W_{down}$) to adapt across iterations while keeping parameters low ($M_i = A_i B_i^T$, with $r\ll d$). Across translation and image classification, RingFormer achieves competitive or superior performance relative to parameter-matched baselines, and analytical tools such as CKA and MAD corroborate closer behavioral alignment to Vanilla Transformer than previous recurrent designs. The work demonstrates that adaptive, low-rank level signals can dramatically improve parameter efficiency without sacrificing core Transformer capabilities, potentially democratizing access to high-performance models. limitations include added compute from the signals and a lack of large-scale language-model experiments, pointing to future work on scaling and broader-domain validation.

Abstract

Transformers have achieved great success in effectively processing sequential data such as text. Their architecture consisting of several attention and feedforward blocks can model relations between elements of a sequence in parallel manner, which makes them very efficient to train and effective in sequence modeling. Even though they have shown strong performance in processing sequential data, the size of their parameters is considerably larger when compared to other architectures such as RNN and CNN based models. Therefore, several approaches have explored parameter sharing and recurrence in Transformer models to address their computational demands. However, such methods struggle to maintain high performance compared to the original transformer model. To address this challenge, we propose our novel approach, RingFormer, which employs one Transformer layer that processes input repeatedly in a circular, ring-like manner, while utilizing low-rank matrices to generate input-dependent level signals. This allows us to reduce the model parameters substantially while maintaining high performance in a variety of tasks such as translation and image classification, as validated in the experiments.

Figures (4)

• Figure 1: Overview of (a) vanilla Transformer vaswani2017attention, (b) recurrent Transformer dehghani2019universal and (d) our RingFormer architecture. The Transformer block represents either encoder or decoder. In the RingFormer model, a single block consisting of attention and feedforward modules is used iteratively, with each sub-modules having unique layer normalization and level signals. (c) illustration of the low-rank matrices representing the level functions, where $M_{down}$ down-projects the input to a lower dimensional space and $M_{up}$ up-projects back to the original space.
• Figure 2: Representation Similarity Analysis using CKA (centered kernel alignment) kornblith2019similarity for the base-size models trained on the translation task. The figures on the upper row are for the encoder part. The figures on the lower row means are for the decoder part. All models have 6 number of layers / iterations. The values on the figures are between 0 and 1, where higher values indicate more similarity of layers between models.
• Figure 3: MAD (mean attention distance) analysis for the models trained on image classification task: ViT, RingFormer, OWF - One Wide FFN, UiT - Universal Transformer. The smaller models shown on the upper row have 8 attention heads, and larger models shown on the lower row have 12 attention heads. The points on the plots represent the mean attention distance of an attention head belonging to a particular Transformer layer.
• Figure 4: Representation Similarity Analysis using CKA (centered kernel alignment) kornblith2019similarity for the large-size models trained on the translation task: Transformer, Ring - RingFormer, OWF - One Wide FFN, Uni - Universal Transformer. The figures on the upper row are for the encoder part. The figures on the lower row means are for the decoder part. All models have 6 number of layers / iterations. The values on the figures are between 0 and 1, where higher values indicate more similarity of layers between models.