Forecasting Future International Events: A Reliable Dataset for Text-Based Event Modeling

TL;DR

This paper introduces WORLDREP (WORLD Relationship and Event Prediction), a novel dataset designed to address limitations in quality and utility in text-based event prediction by leveraging the advanced reasoning capabilities of large-language models (LLMs).

Abstract

Predicting future international events from textual information, such as news articles, has tremendous potential for applications in global policy, strategic decision-making, and geopolitics. However, existing datasets available for this task are often limited in quality, hindering the progress of related research. In this paper, we introduce WORLDREP (WORLD Relationship and Event Prediction), a novel dataset designed to address these limitations by leveraging the advanced reasoning capabilities of large-language models (LLMs). Our dataset features high-quality scoring labels generated through advanced prompt modeling and rigorously validated by domain experts in political science. We showcase the quality and utility of WORLDREP for real-world event prediction tasks, demonstrating its effectiveness through extensive experiments and analysis. Furthermore, we publicly release our dataset along with the full automation source code for data collection, labeling, and benchmarking, aiming to support and advance research in text-based event prediction.

Figures (10)

• Figure 1: A two-stage annotation process using LLMs to extract key countries and analyze their relationships from news articles. The first stage extracts countries mentioned in the news article, and in the second stage, relationships between these countries are labeled. Each stage employs a scratchpad including verification and correction steps to achieve efficient and accurate labeling. These scratchpads are summarized examples and the actual scratchpads and results of each stage can be found in Figure \ref{['main-fig:correcting-samples']} and Appendix \ref{['app-sec:full-scratchpads']}, respectively.
• Figure 2: (a) Distribution of the number of key countries. Our dataset aligns more closely with domain experts compared to GDELT, which tends to identify fewer key countries. (b) Distribution of relationship labels from different sources. The proportions of conflict, cooperation, and unknown labels in our dataset closely match those of domain experts, whereas GDELT lacks the unknown category, leading to significant imbalance.
• Figure 3: Examples of self-correcting in our annotation process: (a) key country extraction and (b) relationship labeling. In (a), the initial list included 'CHN' and 'RUS' incorrectly. 'CHN' was mentioned only as a sea name, and 'RUS' was related to unclear facts. Through self-correcting including verification and explanation, they were removed. In (b), the initial relationship prediction followed the negative tone of the article. The self-correcting process adjusted it to a neutral score, aligning with domain expert opinions.
• Figure 4: An LLM-based prediction framework leveraging past news articles to forecast future relationships between countries. The figure shows the prediction process for the relationship between Israel and Egypt on May 25, 2024. Relevant past articles are highlighted, with summaries provided. The model predicts a conflict relationship (Answer 7), detailing reasons (Answer 1) and a potential specific event (Answer 8) based on the historical context and structured prompt.
• Figure 5: Example of a questionnaire for domain experts for annotation requests.