Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-AR: LLM-powered Automated Reasoning Framework

Rick Chen, Joseph Ternasky, Aaron Ontoyin Yin, Xianling Mu, Fuat Alican, Yigit Ihlamur

TL;DR

LLM-AR addresses the need for reliable, high-precision decisions in high-stakes domains by integrating LLM-generated policies with ProbLog-based automated reasoning. It introduces an iterative policy-generation loop that combines association-rule mining, statistical calibration, and reflection to refine probabilistic rules. On a founder dataset of 6,000 cases, the approach achieves cross-validated precision of 59.5% and recall of 8.7%, about 5.9x the random baseline, while exposing every decision path for human inspection. The framework is explainable, tunable, and generalizable to other high-stakes domains such as medicine, finance and law.

Abstract

Large language models (LLMs) can already identify patterns and reason effectively, yet their variable accuracy hampers adoption in high-stakes decision-making applications. In this paper, we study this issue from a venture capital perspective by predicting idea-stage startup success based on founder traits. (i) To build a reliable prediction model, we introduce LLM-AR, a pipeline inspired by neural-symbolic systems that distils LLM-generated heuristics into probabilistic rules executed by the ProbLog automated-reasoning engine. (ii) An iterative policy-evolution loop incorporates association-rule mining to progressively refine the prediction rules. On unseen folds, LLM-AR achieves 59.5% precision and 8.7% recall, 5.9x the random baseline precision, while exposing every decision path for human inspection. The framework is interpretable and tunable via hyperparameters, showing promise to extend into other domains.

LLM-AR: LLM-powered Automated Reasoning Framework

TL;DR

LLM-AR addresses the need for reliable, high-precision decisions in high-stakes domains by integrating LLM-generated policies with ProbLog-based automated reasoning. It introduces an iterative policy-generation loop that combines association-rule mining, statistical calibration, and reflection to refine probabilistic rules. On a founder dataset of 6,000 cases, the approach achieves cross-validated precision of 59.5% and recall of 8.7%, about 5.9x the random baseline, while exposing every decision path for human inspection. The framework is explainable, tunable, and generalizable to other high-stakes domains such as medicine, finance and law.

Abstract

Large language models (LLMs) can already identify patterns and reason effectively, yet their variable accuracy hampers adoption in high-stakes decision-making applications. In this paper, we study this issue from a venture capital perspective by predicting idea-stage startup success based on founder traits. (i) To build a reliable prediction model, we introduce LLM-AR, a pipeline inspired by neural-symbolic systems that distils LLM-generated heuristics into probabilistic rules executed by the ProbLog automated-reasoning engine. (ii) An iterative policy-evolution loop incorporates association-rule mining to progressively refine the prediction rules. On unseen folds, LLM-AR achieves 59.5% precision and 8.7% recall, 5.9x the random baseline precision, while exposing every decision path for human inspection. The framework is interpretable and tunable via hyperparameters, showing promise to extend into other domains.
Paper Structure (29 sections, 1 equation, 2 figures, 9 tables)

This paper contains 29 sections, 1 equation, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset
  4. Choice of Automated Reasoning System
  5. ProbLog
  6. Methodology
  7. Iterative Training for Policy Generation
  8. Initial policy generation
  9. Statistical policy analysis
  10. Policy reflection
  11. Problog Prediction using Policies
  12. Training-time Policy Evaluation
  13. Final Policy and Hyperparameter Selection
  14. Results
  15. Cross-Validation
  16. ...and 14 more sections

Figures (2)

  • Figure 1: Human-baseline precisions compared with average model precisions obtained by LLMs and LLM-AR on our dataset. Human-baseline precisions are scaled up linearly to reflect the inflation of success rate from the real-world (1.9%) to our dataset (10%).
  • Figure 2: Our LLM-AR pipeline: the iterative training phase produces policies, which are then evaluated by the automated reasoning component. The iterative training phase involves repeated policy generation and reflection based on statistical policy analysis.