Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimal word order for non-causal text generation with Large Language Models: the Spanish case

Andrea Busto-Castiñeira, Silvia García-Méndez, Francisco de Arriba-Pérez, Francisco J. González-Castaño

TL;DR

This work investigates whether non-causal word generation can improve Spanish NLG by estimating the maximum-likelihood generation order using a Viterbi-based approach. It compares non-causal generation with decoder-only causal generation on a Spanish dataset, employing RoBERTa (non-causal) and GPT-2/GPT-3 (causal) models. The findings show that non-causal ordering generally yields higher sentence probabilities, with the relationship to causal order being highly dependent on syntactic structure, and reveal systematic patterns (e.g., object–subject–verb tendencies) in optimal orders. The results suggest that leveraging non-causal generation can enrich Spanish syntax in NLG and motivate further development of order-estimation and reinforcement-learning strategies for multilingual contexts, including future work on more complex constructions and cross-language extensions.

Abstract

Natural Language Generation (NLG) popularity has increased owing to the progress in Large Language Models (LLMs), with zero-shot inference capabilities. However, most neural systems utilize decoder-only causal (unidirectional) transformer models, which are effective for English but may reduce the richness of languages with less strict word order, subject omission, or different relative clause attachment preferences. This is the first work that analytically addresses optimal text generation order for non-causal language models. We present a novel Viterbi algorithm-based methodology for maximum likelihood word order estimation. We analyze the non-causal most-likelihood order probability for NLG in Spanish and, then, the probability of generating the same phrases with Spanish causal NLG. This comparative analysis reveals that causal NLG prefers English-like SVO structures. We also analyze the relationship between optimal generation order and causal left-to-right generation order using Spearman's rank correlation. Our results demonstrate that the ideal order predicted by the maximum likelihood estimator is not closely related to the causal order and may be influenced by the syntactic structure of the target sentence.

Optimal word order for non-causal text generation with Large Language Models: the Spanish case

TL;DR

This work investigates whether non-causal word generation can improve Spanish NLG by estimating the maximum-likelihood generation order using a Viterbi-based approach. It compares non-causal generation with decoder-only causal generation on a Spanish dataset, employing RoBERTa (non-causal) and GPT-2/GPT-3 (causal) models. The findings show that non-causal ordering generally yields higher sentence probabilities, with the relationship to causal order being highly dependent on syntactic structure, and reveal systematic patterns (e.g., object–subject–verb tendencies) in optimal orders. The results suggest that leveraging non-causal generation can enrich Spanish syntax in NLG and motivate further development of order-estimation and reinforcement-learning strategies for multilingual contexts, including future work on more complex constructions and cross-language extensions.

Abstract

Natural Language Generation (NLG) popularity has increased owing to the progress in Large Language Models (LLMs), with zero-shot inference capabilities. However, most neural systems utilize decoder-only causal (unidirectional) transformer models, which are effective for English but may reduce the richness of languages with less strict word order, subject omission, or different relative clause attachment preferences. This is the first work that analytically addresses optimal text generation order for non-causal language models. We present a novel Viterbi algorithm-based methodology for maximum likelihood word order estimation. We analyze the non-causal most-likelihood order probability for NLG in Spanish and, then, the probability of generating the same phrases with Spanish causal NLG. This comparative analysis reveals that causal NLG prefers English-like SVO structures. We also analyze the relationship between optimal generation order and causal left-to-right generation order using Spearman's rank correlation. Our results demonstrate that the ideal order predicted by the maximum likelihood estimator is not closely related to the causal order and may be influenced by the syntactic structure of the target sentence.

Paper Structure

This paper contains 13 sections, 8 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Causality in generative transformer language models
  4. Viterbi algorithm in nlp
  5. Methodology
  6. Language modeling and sentence probability
  7. Viterbi maximum likelihood generation order estimation
  8. Spearman's rank correlation coefficient
  9. Results
  10. Language models
  11. Data set
  12. Results and discussion
  13. Conclusions

Figures (7)

  • Figure 1: Viterbi for maximum likelihood word ordering estimation, toy example with $N=3$. The optimal Viterbi path is in red.
  • Figure 2: Optimal-causal generation probability ratio in logarithmic units from declarative sentences
  • Figure 3: Optimal-causal generation probability ratio in logarithmic units for interrogative sentences
  • Figure 4: Histogram of Spearman's rank correlation coefficient ($\rho$) for declarative sentences.
  • Figure 5: Histogram of Spearman's rank correlation coefficient ($\rho$) by syntactic structure for declarative sentences. (a) svo. (b) sov. (c) vso. (d) vos. (e) osv. (f) ovs.
  • ...and 2 more figures