Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks

Mirelle Bueno, Roberto Lotufo, Rodrigo Nogueira

TL;DR

MLissard is introduced, a multilingual benchmark designed to evaluate models’ abilities to process and generate texts of varied lengths and offers a mechanism for controlling sequence complexity.

Abstract

Language models are now capable of solving tasks that require dealing with long sequences consisting of hundreds of thousands of tokens. However, they often fail on tasks that require repetitive use of simple rules, even on sequences that are much shorter than those seen during training. For example, state-of-the-art LLMs can find common items in two lists with up to 20 items but fail when lists have 80 items. In this paper, we introduce MLissard, a multilingual benchmark designed to evaluate models' abilities to process and generate texts of varied lengths and offers a mechanism for controlling sequence complexity. Our evaluation of open-source and proprietary models show a consistent decline in performance across all models and languages as the complexity of the sequence increases. Surprisingly, the use of in-context examples in languages other than English helps increase extrapolation performance significantly. The datasets and code are available at https://github.com/unicamp-dl/Lissard

MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks

TL;DR

MLissard is introduced, a multilingual benchmark designed to evaluate models’ abilities to process and generate texts of varied lengths and offers a mechanism for controlling sequence complexity.

Abstract

Language models are now capable of solving tasks that require dealing with long sequences consisting of hundreds of thousands of tokens. However, they often fail on tasks that require repetitive use of simple rules, even on sequences that are much shorter than those seen during training. For example, state-of-the-art LLMs can find common items in two lists with up to 20 items but fail when lists have 80 items. In this paper, we introduce MLissard, a multilingual benchmark designed to evaluate models' abilities to process and generate texts of varied lengths and offers a mechanism for controlling sequence complexity. Our evaluation of open-source and proprietary models show a consistent decline in performance across all models and languages as the complexity of the sequence increases. Surprisingly, the use of in-context examples in languages other than English helps increase extrapolation performance significantly. The datasets and code are available at https://github.com/unicamp-dl/Lissard
Paper Structure (15 sections, 5 figures, 4 tables)

This paper contains 15 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Datasets Description
  4. Key entities
  5. Tasks
  6. Object Counting
  7. List Intersection
  8. Last Letter Concatenation
  9. Repeat Copy Logic
  10. Baseline Methods
  11. Results
  12. Impact of model size
  13. Can cross language improve extrapolation performance?
  14. Conclusion
  15. Limitations

Figures (5)

  • Figure 1: Performance of GPT-4 on the MLissard benchmark. See Table \ref{['tab:key_entity_bins']} for the definition of the bins.
  • Figure 2: Template for evaluation. Being (a) Instruction and examples of tasks in the target language; (b) Instruction in the target language and multilingual examples.
  • Figure 3: GPT-4 performance in the MLissard.
  • Figure 4: Comparison of Llama-3.1-405B vs. GPT-4 performance in the MLissard Benchmark
  • Figure 5: Average accuracy considering all bins. Since (1) Baseline - Both the instruction and the examples derive from the same target language; (2) instruction in the language that performed better or worse and a examples in the target language; (3) Instruction in target language and multilingual examples.