Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Language Models Represent the Past without Anachronism?

Ted Underwood, Laura K. Nelson, Matthew Wilkens

TL;DR

The paper probes whether language models can represent the past without anachronism, comparing prompting with period examples to fine-tuning on period prose. Prompting alone fails to produce authentically period-consistent outputs, while fine-tuning on a few hundred in-period passages yields stylistically plausible text that confounds automated dating but remains detectable to human readers. A period-pretrained model (GPT-1914) can be anachronism-free, yet its fluency and scale are limited and expensive, raising questions about practicality. Overall, the work suggests that simple prompting is insufficient, fine-tuning helps, but robust, low-anachronism historical modeling may require substantial pretraining on historical data or new benchmarking approaches, balancing authenticity and fluency for social research.

Abstract

Before researchers can use language models to simulate the past, they need to understand the risk of anachronism. We find that prompting a contemporary model with examples of period prose does not produce output consistent with period style. Fine-tuning produces results that are stylistically convincing enough to fool an automated judge, but human evaluators can still distinguish fine-tuned model outputs from authentic historical text. We tentatively conclude that pretraining on period prose may be required in order to reliably simulate historical perspectives for social research.

Can Language Models Represent the Past without Anachronism?

TL;DR

The paper probes whether language models can represent the past without anachronism, comparing prompting with period examples to fine-tuning on period prose. Prompting alone fails to produce authentically period-consistent outputs, while fine-tuning on a few hundred in-period passages yields stylistically plausible text that confounds automated dating but remains detectable to human readers. A period-pretrained model (GPT-1914) can be anachronism-free, yet its fluency and scale are limited and expensive, raising questions about practicality. Overall, the work suggests that simple prompting is insufficient, fine-tuning helps, but robust, low-anachronism historical modeling may require substantial pretraining on historical data or new benchmarking approaches, balancing authenticity and fluency for social research.

Abstract

Before researchers can use language models to simulate the past, they need to understand the risk of anachronism. We find that prompting a contemporary model with examples of period prose does not produce output consistent with period style. Fine-tuning produces results that are stylistically convincing enough to fool an automated judge, but human evaluators can still distinguish fine-tuned model outputs from authentic historical text. We tentatively conclude that pretraining on period prose may be required in order to reliably simulate historical perspectives for social research.
Paper Structure (10 sections, 4 figures, 2 tables)

This paper contains 10 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Prompting with examples from the period
  3. GPT-1914: A pretrained historical language model
  4. Evaluating in-context learning
  5. Fine-tuning on examples from the period
  6. Automatic evaluation by a date-prediction model
  7. Human evaluation
  8. First evaluation strategy: direct judgment
  9. Second evaluation strategy: intruder test
  10. Conclusions

Figures (4)

  • Figure 1: Example of a prompted continuation by GPT-4o.
  • Figure 2: Example of a continuation by GPT-1914, responding to the same passage provided in figure \ref{['fig:promptedcontinuation']}.
  • Figure 3: The stylistic center of mass of prompted and period-pretrained models. Kernel density plots for ground truth passages drawn from books, 1905--1914, as well as continuations of those passages by GPT-4o (1-shot and 20-shot) and GPT-1914 (trained only on text 1880--1914).
  • Figure 4: The stylistic center of mass of fine-tuned and period-pretrained models. Kernel density plots for ground truth passages drawn from books 1905--1914, as well as continuations of those passages by GPT-1914 and by a version of GPT-4o-mini fine-tuned on passages 1905--1914.