Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bidirectional Awareness Induction in Autoregressive Seq2Seq Models

Jia Cheng Hu, Roberto Cavicchioli, Alessandro Capotondi

TL;DR

This paper tackles the limitations of autoregressive Seq2Seq models by introducing Bidirectional Awareness Induction (BAI), which leverages Pivot elements trained with a bidirectional objective during training while preserving autoregressive decoding. By applying BAI to Transformer, ExpansionNet v2, and GPT-2 across Image Captioning, Neural Machine Translation, and Text Summarization, the approach achieves consistent improvements without architectural changes and even benefits pre-trained models. The key ideas are Pivot Selection, Length Equalization, and Decoder sequence reconstruction, with a joint objective combining the standard cross-entropy loss and a Mean Square bidirectional loss. The work demonstrates that targeted, training-only bidirectional cues can enhance intermediate representations and performance, offering a practical path to boosting seq2seq systems in both research and deployed large-language-model contexts.

Abstract

Autoregressive Sequence-To-Sequence models are the foundation of many Deep Learning achievements in major research fields such as Vision and Natural Language Processing. Despite that, they still present significant limitations. For instance, when errors occur in the early steps of the prediction, the whole output is severely affected. Such reliance on previously predicted tokens and the inherent computational unfriendliness of sequential algorithms, motivated researchers to explore different architectures and methods in the search for bidirectional approaches. In this work, we introduce the Bidirectional Awareness Induction (BAI), a training method that leverages a subset of elements in the network, the Pivots, to perform bidirectional learning without breaking the autoregressive constraints. To showcase its flexibility, we apply the method to three architectures, the Transformer, ExpansionNet v2 and GPT, then perform experiments over three tasks. Experimental results showcase BAI's effectiveness on all selected tasks and architectures. In particular, we observed an increase of up to 2.4 CIDEr in Image-Captioning, 4.96 BLEU in Neural Machine Translation, and 1.16 ROUGE in Text Summarization compared to the respective baselines. Notably, BAI not only has a positive impact on models trained from scratch but on pre-trained models as well. Such an aspect, combined with the absence of architectural requirements synergizes well with the current trend of LLMs.

Bidirectional Awareness Induction in Autoregressive Seq2Seq Models

TL;DR

This paper tackles the limitations of autoregressive Seq2Seq models by introducing Bidirectional Awareness Induction (BAI), which leverages Pivot elements trained with a bidirectional objective during training while preserving autoregressive decoding. By applying BAI to Transformer, ExpansionNet v2, and GPT-2 across Image Captioning, Neural Machine Translation, and Text Summarization, the approach achieves consistent improvements without architectural changes and even benefits pre-trained models. The key ideas are Pivot Selection, Length Equalization, and Decoder sequence reconstruction, with a joint objective combining the standard cross-entropy loss and a Mean Square bidirectional loss. The work demonstrates that targeted, training-only bidirectional cues can enhance intermediate representations and performance, offering a practical path to boosting seq2seq systems in both research and deployed large-language-model contexts.

Abstract

Autoregressive Sequence-To-Sequence models are the foundation of many Deep Learning achievements in major research fields such as Vision and Natural Language Processing. Despite that, they still present significant limitations. For instance, when errors occur in the early steps of the prediction, the whole output is severely affected. Such reliance on previously predicted tokens and the inherent computational unfriendliness of sequential algorithms, motivated researchers to explore different architectures and methods in the search for bidirectional approaches. In this work, we introduce the Bidirectional Awareness Induction (BAI), a training method that leverages a subset of elements in the network, the Pivots, to perform bidirectional learning without breaking the autoregressive constraints. To showcase its flexibility, we apply the method to three architectures, the Transformer, ExpansionNet v2 and GPT, then perform experiments over three tasks. Experimental results showcase BAI's effectiveness on all selected tasks and architectures. In particular, we observed an increase of up to 2.4 CIDEr in Image-Captioning, 4.96 BLEU in Neural Machine Translation, and 1.16 ROUGE in Text Summarization compared to the respective baselines. Notably, BAI not only has a positive impact on models trained from scratch but on pre-trained models as well. Such an aspect, combined with the absence of architectural requirements synergizes well with the current trend of LLMs.
Paper Structure (16 sections, 5 equations, 2 figures, 6 tables)

This paper contains 16 sections, 5 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Bidirectional Awareness Induction (BAI)
  5. BAI in Transformer
  6. BAI in ExpansionNet v2
  7. BAI in Large Pre-Trained Models
  8. Experimental Setup
  9. Dataset
  10. Training and Models Details
  11. Evaluation Methods
  12. Results
  13. BAI Results
  14. BAI Against Other Bidirectionality Approaches
  15. BAI Weight Term Impact
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Pivot selection in case of Transformer vaswani2017attention, ExpansionNet v2 hu2023exploiting and GPT-2 radford2019language architectures. Pivots are highlighted in red colour, processing layers are depicted in blue.
  • Figure 2: BAI weight function impact on Image Captioning performances. a) Depiction of several strategies of weight term functions. $\Lambda1(t)$=$10^{-3}$, $\Lambda2(t)$=$10^{-6}$, $\Lambda3(t)$=$1$, $\Lambda4(t)$=$10^{-6}$$+$$(1-10^{-6})$$\cdot$$(t/30)$, $\Lambda5(t)$=$(1-10^{-6})$$\cdot$$(30-t)/30$$+$$10^{-6}$, and $\Lambda^*(t)$ is the function described in Section \ref{['sec:training_and_models']}. $t$ denotes the epoch. b) Validation score of the baseline architecture (ExpansionNet v2) without BAI. The red line denotes the point where the model achieves the highest score. c) Best CIDEr-D score observed from different selections of $\Lambda$.