STRUX: An LLM for Decision-Making with Structured Explanations

TL;DR

This paper introduces a new LLM decision-making framework called STRUX, which enhances LLM decision-making by providing structured explanations that include favorable and adverse facts related to the decision, along with their respective strengths.

Abstract

Countless decisions shape our daily lives, and it is paramount to understand the how and why behind these choices. In this paper, we introduce a new LLM decision-making framework called STRUX, which enhances LLM decision-making by providing structured explanations. These include favorable and adverse facts related to the decision, along with their respective strengths. STRUX begins by distilling lengthy information into a concise table of key facts. It then employs a series of self-reflection steps to determine which of these facts are pivotal, categorizing them as either favorable or adverse in relation to a specific decision. Lastly, we fine-tune an LLM to identify and prioritize these key facts to optimize decision-making. STRUX has been evaluated on the challenging task of forecasting stock investment decisions based on earnings call transcripts and demonstrated superior performance against strong baselines. It enhances decision transparency by allowing users to understand the impact of different factors, representing a meaningful step towards practical decision-making with LLMs.

Figures (7)

• Figure 1: STRUX's explanations consist of three components: {supporting facts, a decision, and a brief justification}. Supporting facts can include both positive (green) and negative (red) aspects, along with their strengths.
• Figure 2: Each iteration of self-reflection improves the accuracy of decision-making. We show the percentage of training instances that receive correct decisions after each iteration. Our STRUX model is instructed to select from three ranges of supporting facts: 3-6, 6-10, and 10-15. The selection of 6-10 supporting facts consistently yielded the highest accuracy.
• Figure 3: Confusion matrix after each reflection.
• Figure 4: We input executive speeches from the Prepared Remarks or Q&A sessions into the LLM. Summaries are proportional in input length. Each speech from the Prepared Remarks is summarized into 3-5 key facts, while those from the Q&A session are condensed into 1-3 key facts. Fact tables are generated using gpt-4o-mini-2024-07-18.
• Figure 5: STRUX is tasked with predicting a company's post-earnings stock trend to inform the investment decision. It is set to select the most relevant facts from a provided fact table, ensuring a balanced representation of positive and negative facts affecting the stock price. Each selected fact is evaluated for its potential impact on the stock's price movement. A "+" symbol indicates a positive impact, with the number of symbols varying from one (+) to three (+++) showing the increasing strength. Conversely, a "-" symbol denotes a negative impact, with one (-) to three (---) symbols reflecting the severity of the negative influence. The system then analyzes all the selected facts to forecast the direction of the stock price movement. The outcomes include: Strongly Buy (SB), Buy (B), Hold (H), Sell (S), or Strongly Sell (SS). It also provides a justification elaborating on its rationale, focusing on the key facts that influence this decision. Additionally, we tested gpt-4o-mini-2024-07-18 and Llama3-8b-Instruct using this prompt by providing either full transcripts or concise fact tables to elicit investment decisions.