Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A three-step machine learning approach to predict market bubbles with financial news

Abraham Atsiwo

TL;DR

The paper tackles predicting market bubbles in the S&P 500 by integrating macroeconomic indicators with financial-news sentiment in a three-step pipeline. It combines a real-time bubble detector (PSY via GSADF), finbert-lc based sentiment analysis, and ensemble predictors (e.g., XGBoost) to forecast bubble occurrences. Empirical results show macro variables such as GDP, BOP, and CPI are the primary drivers, while sentiment adds limited incremental value; XGBoost offers the best balance of precision and recall. The work provides an actionable early-warning framework for investors, regulators, and policymakers to mitigate systemic risk, with potential extensions to richer sentiment modeling and cross-market analyses.

Abstract

This study presents a three-step machine learning framework to predict bubbles in the S&P 500 stock market by combining financial news sentiment with macroeconomic indicators. Building on traditional econometric approaches, the proposed approach predicts bubble formation by integrating textual and quantitative data sources. In the first step, bubble periods in the S&P 500 index are identified using a right-tailed unit root test, a widely recognized real-time bubble detection method. The second step extracts sentiment features from large-scale financial news articles using natural language processing (NLP) techniques, which capture investors' expectations and behavioral patterns. In the final step, ensemble learning methods are applied to predict bubble occurrences based on high sentiment-based and macroeconomic predictors. Model performance is evaluated through k-fold cross-validation and compared against benchmark machine learning algorithms. Empirical results indicate that the proposed three-step ensemble approach significantly improves predictive accuracy and robustness, providing valuable early warning insights for investors, regulators, and policymakers in mitigating systemic financial risks.

A three-step machine learning approach to predict market bubbles with financial news

TL;DR

The paper tackles predicting market bubbles in the S&P 500 by integrating macroeconomic indicators with financial-news sentiment in a three-step pipeline. It combines a real-time bubble detector (PSY via GSADF), finbert-lc based sentiment analysis, and ensemble predictors (e.g., XGBoost) to forecast bubble occurrences. Empirical results show macro variables such as GDP, BOP, and CPI are the primary drivers, while sentiment adds limited incremental value; XGBoost offers the best balance of precision and recall. The work provides an actionable early-warning framework for investors, regulators, and policymakers to mitigate systemic risk, with potential extensions to richer sentiment modeling and cross-market analyses.

Abstract

This study presents a three-step machine learning framework to predict bubbles in the S&P 500 stock market by combining financial news sentiment with macroeconomic indicators. Building on traditional econometric approaches, the proposed approach predicts bubble formation by integrating textual and quantitative data sources. In the first step, bubble periods in the S&P 500 index are identified using a right-tailed unit root test, a widely recognized real-time bubble detection method. The second step extracts sentiment features from large-scale financial news articles using natural language processing (NLP) techniques, which capture investors' expectations and behavioral patterns. In the final step, ensemble learning methods are applied to predict bubble occurrences based on high sentiment-based and macroeconomic predictors. Model performance is evaluated through k-fold cross-validation and compared against benchmark machine learning algorithms. Empirical results indicate that the proposed three-step ensemble approach significantly improves predictive accuracy and robustness, providing valuable early warning insights for investors, regulators, and policymakers in mitigating systemic financial risks.
Paper Structure (11 sections, 15 equations, 8 figures, 6 tables)

This paper contains 11 sections, 15 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Methodology: The Three-Stage Analysis
  3. The PSY Procedure for Bubble Detection
  4. Financial Sentiment Analysis with finbert-lc
  5. Ensemble Method for Robust Prediction
  6. Experimental Results
  7. Data Sources and Processing
  8. Evaluation Metrics and Results
  9. Feature Importance and Model Diagnostic, RQ1
  10. Varying Threshold and Where the Model Fails, RQ3
  11. Conclusion and Further Work

Figures (8)

  • Figure 1: Boosting algorithm. Base learners are trained on weighted samples, where each base learner correct the errors of the previous learner.
  • Figure 2: Bagging algorithm. Bootstrap samples are generated from the original sample and $m$ classifiers are trained independently.
  • Figure 3: A decision tree hierarchy from the root node to the terminal node
  • Figure 4: S&P 500 price-dividend ratio with detected bubble periods drawn with vertical lines for monthly data
  • Figure 5: Feature Importance of the Average of Ensemble Methods
  • ...and 3 more figures