MuLD: The Multitask Long Document Benchmark
G Thomas Hudson, Noura Al Moubayed
TL;DR
MuLD tackles the gap in evaluation for long-document understanding by introducing a multitask benchmark where each input exceeds 10,000 tokens. It adaptively extends existing tasks across six real-world domains to stress long-range dependencies and tests both standard (T5) and efficient (Longformer) transformers with chunking baselines. The results show that longer-context models generally outperform standard ones, underscoring the importance of extended context for long documents, while some tasks reveal limitations in current approaches and the need for improved chunking strategies. By releasing data and code, MuLD provides a foundation for measuring progress and guiding future research toward scalable, long-range NLP models with practical impact on document-rich applications.
Abstract
The impressive progress in NLP techniques has been driven by the development of multi-task benchmarks such as GLUE and SuperGLUE. While these benchmarks focus on tasks for one or two input sentences, there has been exciting work in designing efficient techniques for processing much longer inputs. In this paper, we present MuLD: a new long document benchmark consisting of only documents over 10,000 tokens. By modifying existing NLP tasks, we create a diverse benchmark which requires models to successfully model long-term dependencies in the text. We evaluate how existing models perform, and find that our benchmark is much more challenging than their `short document' equivalents. Furthermore, by evaluating both regular and efficient transformers, we show that models with increased context length are better able to solve the tasks presented, suggesting that future improvements in these models are vital for solving similar long document problems. We release the data and code for baselines to encourage further research on efficient NLP models.