Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Future Language Modeling from Temporal Document History

Changmao Li, Jeffrey Flanigan

TL;DR

This paper formalizes the task of future language modeling, defining generation of future texts conditioned on a temporal history of documents. It introduces three models that inject temporal information into a pre-trained language model via a learned temporal bias: a Word Frequency Model, a Temporal Contextual Model, and a Doubly Contextualized Model with gating to balance temporal cues and in-context generation. Evaluations on ACL abstracts show that temporal models outperform non-temporal baselines on automatic metrics (PPL, CPL, CM) and human judgments, with the DTVR variant delivering the strongest performance and coherent, timely content. The work demonstrates the viability of predicting future textual content from historical data and provides a framework for extending temporal text generation to other domains and architectures.

Abstract

Predicting the future is of great interest across many aspects of human activity. Businesses are interested in future trends, traders are interested in future stock prices, and companies are highly interested in future technological breakthroughs. While there are many automated systems for predicting future numerical data, such as weather, stock prices, and demand for products, there is relatively little work in automatically predicting textual data. Humans are interested in textual data predictions because it is a natural format for our consumption, and experts routinely make predictions in a textual format (Christensen et al., 2004; Tetlock & Gardner, 2015; Frick, 2015). However, there has been relatively little formalization of this general problem in the machine learning or natural language processing communities. To address this gap, we introduce the task of future language modeling: probabilistic modeling of texts in the future based on a temporal history of texts. To our knowledge, our work is the first work to formalize the task of predicting the future in this way. We show that it is indeed possible to build future language models that improve upon strong non-temporal language model baselines, opening the door to working on this important, and widely applicable problem.

Future Language Modeling from Temporal Document History

TL;DR

This paper formalizes the task of future language modeling, defining generation of future texts conditioned on a temporal history of documents. It introduces three models that inject temporal information into a pre-trained language model via a learned temporal bias: a Word Frequency Model, a Temporal Contextual Model, and a Doubly Contextualized Model with gating to balance temporal cues and in-context generation. Evaluations on ACL abstracts show that temporal models outperform non-temporal baselines on automatic metrics (PPL, CPL, CM) and human judgments, with the DTVR variant delivering the strongest performance and coherent, timely content. The work demonstrates the viability of predicting future textual content from historical data and provides a framework for extending temporal text generation to other domains and architectures.

Abstract

Predicting the future is of great interest across many aspects of human activity. Businesses are interested in future trends, traders are interested in future stock prices, and companies are highly interested in future technological breakthroughs. While there are many automated systems for predicting future numerical data, such as weather, stock prices, and demand for products, there is relatively little work in automatically predicting textual data. Humans are interested in textual data predictions because it is a natural format for our consumption, and experts routinely make predictions in a textual format (Christensen et al., 2004; Tetlock & Gardner, 2015; Frick, 2015). However, there has been relatively little formalization of this general problem in the machine learning or natural language processing communities. To address this gap, we introduce the task of future language modeling: probabilistic modeling of texts in the future based on a temporal history of texts. To our knowledge, our work is the first work to formalize the task of predicting the future in this way. We show that it is indeed possible to build future language models that improve upon strong non-temporal language model baselines, opening the door to working on this important, and widely applicable problem.
Paper Structure (25 sections, 13 equations, 3 figures, 6 tables, 1 algorithm)

This paper contains 25 sections, 13 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Task Overview
  3. Approach
  4. Overview of Models
  5. The Word Frequency Model
  6. The Temporal Contextual Model
  7. The Doubly Contextualized Model
  8. Experiments: Future Abstract Prediction
  9. Dataset processing
  10. Models
  11. Hyperparameter Settings
  12. Automatic Evaluation Metrics
  13. Automatic Evaluation Results
  14. Human Evaluation
  15. Case Study
  16. ...and 10 more sections

Figures (3)

  • Figure 1: (a) represents an example showing how abstracts in recent history are related to the future. In this example, the text of the abstract of the RoBERTa paper Liu2019 anticipates the rise of papers about "language model pretraining" du-etal-2022-glmbhattacharjee-etal-2022-banglabertchi-etal-2022-xlm. (b) shows the word frequencies by year in NLP abstracts for some representative words, which reflects topic/approach changes over the years, i.e., "pretrain" started to dramatically go up after 2018 because of BERT, and "neural" became popular after 2013 because of deep learning.
  • Figure 2: Our proposed models.
  • Figure 3: # of abstracts by year