Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Forecasting Future Language: Context Design for Mention Markets

Sumin Kim, Jihoon Kwon, Yoon Kim, Nicole Kagan, Raffi Khatchadourian, Wonbin Ahn, Alejandro Lopez-Lira, Jaewon Lee, Yoontae Hwang, Oscar Levy, Yongjae Lee, Chanyeol Choi

TL;DR

Three insights are found: richer context consistently improves forecasting performance; market-conditioned prompting (MCP), which treats the market probability as a prior and updates it using textual evidence, yields better-calibrated forecasts; and a mixture of the market probability and MCP outperforms the market baseline.

Abstract

Mention markets, a type of prediction market in which contracts resolve based on whether a specified keyword is mentioned during a future public event, require accurate probabilistic forecasts of keyword-mention outcomes. While recent work shows that large language models (LLMs) can generate forecasts competitive with human forecasters, it remains unclear how input context should be designed to support accurate prediction. In this paper, we study this question through experiments on earnings-call mention markets, which require forecasting whether a company will mention a specified keyword during its upcoming call. We run controlled comparisons varying (i) which contextual information is provided (news and/or prior earnings-call transcripts) and (ii) how \textit{market probability}, (i.e., prediction market contract price) is used. We introduce Market-Conditioned Prompting (MCP), which explicitly treats the market-implied probability as a prior and instructs the LLM to update this prior using textual evidence, rather than re-predicting the base rate from scratch. In our experiments, we find three insights: (1) richer context consistently improves forecasting performance; (2) market-conditioned prompting (MCP), which treats the market probability as a prior and updates it using textual evidence, yields better-calibrated forecasts; and (3) a mixture of the market probability and MCP (MixMCP) outperforms the market baseline. By dampening the LLM's posterior update with the market prior, MixMCP yields more robust predictions than either the market or the LLM alone.

Forecasting Future Language: Context Design for Mention Markets

TL;DR

Three insights are found: richer context consistently improves forecasting performance; market-conditioned prompting (MCP), which treats the market probability as a prior and updates it using textual evidence, yields better-calibrated forecasts; and a mixture of the market probability and MCP outperforms the market baseline.

Abstract

Mention markets, a type of prediction market in which contracts resolve based on whether a specified keyword is mentioned during a future public event, require accurate probabilistic forecasts of keyword-mention outcomes. While recent work shows that large language models (LLMs) can generate forecasts competitive with human forecasters, it remains unclear how input context should be designed to support accurate prediction. In this paper, we study this question through experiments on earnings-call mention markets, which require forecasting whether a company will mention a specified keyword during its upcoming call. We run controlled comparisons varying (i) which contextual information is provided (news and/or prior earnings-call transcripts) and (ii) how \textit{market probability}, (i.e., prediction market contract price) is used. We introduce Market-Conditioned Prompting (MCP), which explicitly treats the market-implied probability as a prior and instructs the LLM to update this prior using textual evidence, rather than re-predicting the base rate from scratch. In our experiments, we find three insights: (1) richer context consistently improves forecasting performance; (2) market-conditioned prompting (MCP), which treats the market probability as a prior and updates it using textual evidence, yields better-calibrated forecasts; and (3) a mixture of the market probability and MCP (MixMCP) outperforms the market baseline. By dampening the LLM's posterior update with the market prior, MixMCP yields more robust predictions than either the market or the LLM alone.
Paper Structure (28 sections, 6 equations, 3 figures, 6 tables)

This paper contains 28 sections, 6 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Earnings-Call Mention Markets.
  4. Forecasting Task.
  5. Evaluation Metrics.
  6. Methods
  7. Inputs and Base Forecaster.
  8. Textual Context.
  9. Market Signal as Plain Text.
  10. Market-Conditioned Prompting (MCP).
  11. Mixture Forecast (MixMCP).
  12. Experiments
  13. Experimental Setup
  14. Dataset.
  15. Input Construction.
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Resolving Market Ambiguity with MixMCP. The diagram illustrates a mid-confidence scenario where the market probability is uncertain (45%). By conditioning on relevant textual evidence (e.g., earnings transcripts, news), MCP identifies a strong signal ($p_{\text{mcp}} = 80\%$) that the market has not fully priced in. Finally, MixMCP combines the stable market prior with the LLM's evidence-based update ($\alpha = 0.7$), producing a refined forecast (55.5%) that correctly anticipates the resolution.
  • Figure 3: Shared user input template used across variants.
  • Figure 4: Disagreement analysis between the market probability baseline and MCP. Disagree $n$ denotes instances where the two methods differ in binary prediction. Columns report the count of instances where each method achieves a lower Brier score.