Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Large Language Models do Analytical Reasoning?

Yebowen Hu, Kaiqiang Song, Sangwoo Cho, Xiaoyang Wang, Hassan Foroosh, Dong Yu, Fei Liu

TL;DR

It is concluded that task complexity depends on the length of context, the information density, and the presence of related information.

Abstract

This paper explores the cutting-edge Large Language Model with analytical reasoning on sports. Our analytical reasoning embodies the tasks of letting large language models count how many points each team scores in a quarter in the NBA and NFL games. Our major discoveries are in two folds. Firstly, we find among all the models we employed, GPT-4 stands out in effectiveness, followed by Claude-2.1, with GPT-3.5, Gemini-Pro, and Llama-2-70b lagging behind. Specifically, we compare three different prompting techniques and a divide-and-conquer approach, we find that the latter was the most effective. Our divide-and-conquer approach breaks down play-by-play data into smaller, more manageable segments, solves each piece individually, and then aggregates them together. Besides the divide-and-conquer approach, we also explore the Chain of Thought (CoT) strategy, which markedly improves outcomes for certain models, notably GPT-4 and Claude-2.1, with their accuracy rates increasing significantly. However, the CoT strategy has negligible or even detrimental effects on the performance of other models like GPT-3.5 and Gemini-Pro. Secondly, to our surprise, we observe that most models, including GPT-4, struggle to accurately count the total scores for NBA quarters despite showing strong performance in counting NFL quarter scores. This leads us to further investigate the factors that impact the complexity of analytical reasoning tasks with extensive experiments, through which we conclude that task complexity depends on the length of context, the information density, and the presence of related information. Our research provides valuable insights into the complexity of analytical reasoning tasks and potential directions for developing future large language models.

Can Large Language Models do Analytical Reasoning?

TL;DR

It is concluded that task complexity depends on the length of context, the information density, and the presence of related information.

Abstract

This paper explores the cutting-edge Large Language Model with analytical reasoning on sports. Our analytical reasoning embodies the tasks of letting large language models count how many points each team scores in a quarter in the NBA and NFL games. Our major discoveries are in two folds. Firstly, we find among all the models we employed, GPT-4 stands out in effectiveness, followed by Claude-2.1, with GPT-3.5, Gemini-Pro, and Llama-2-70b lagging behind. Specifically, we compare three different prompting techniques and a divide-and-conquer approach, we find that the latter was the most effective. Our divide-and-conquer approach breaks down play-by-play data into smaller, more manageable segments, solves each piece individually, and then aggregates them together. Besides the divide-and-conquer approach, we also explore the Chain of Thought (CoT) strategy, which markedly improves outcomes for certain models, notably GPT-4 and Claude-2.1, with their accuracy rates increasing significantly. However, the CoT strategy has negligible or even detrimental effects on the performance of other models like GPT-3.5 and Gemini-Pro. Secondly, to our surprise, we observe that most models, including GPT-4, struggle to accurately count the total scores for NBA quarters despite showing strong performance in counting NFL quarter scores. This leads us to further investigate the factors that impact the complexity of analytical reasoning tasks with extensive experiments, through which we conclude that task complexity depends on the length of context, the information density, and the presence of related information. Our research provides valuable insights into the complexity of analytical reasoning tasks and potential directions for developing future large language models.
Paper Structure (15 sections, 1 equation, 10 figures, 6 tables)

This paper contains 15 sections, 1 equation, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Data & Task
  5. Applied Models
  6. Applied Methods
  7. Experiments
  8. Analysis
  9. Length Factor
  10. Information Density Factor
  11. Related Information Factor
  12. Conclusion
  13. Appendix
  14. Hallucinated Responses
  15. Instruction Following Verification

Figures (10)

  • Figure 1: Play-by-play descriptions of an NBA game, including timestamps, play-by-play descriptions, team affiliations and total points. The content indicated in dot circles are inputs for our task. The total points for both teams are withheld from LLMs. Highlighted descriptions are scoring moves originally labeled in source data. This image was created with the assistance of DALL·E
  • Figure 2: We conducted comparative experiments on [A] three instructions as figure presents. After instruction, We attach sports content in order of [B] initial scores, [C] team player table and [D] game input. Aggregating the entire user message as our task instance.
  • Figure 3: A workflow depict the divide-and-conquer approach at step-size of three. We truncate a quarter of the play-by-play(PBP) description into segments then bind with natural language instruction as input to LLMs. Finally, all responses are tallied to calculate the quarter scores.
  • Figure 4: Length Analysis on NBA data
  • Figure 5: Density analysis on NFL data
  • ...and 5 more figures