Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Large Language Models Effectively Process and Execute Financial Trading Instructions?

Yu Kang, Ge Wang, Xin Yang, Yuda Wang, Mingwen Liu

TL;DR

This work tackles the challenge of translating natural-language financial trading instructions into executable commands. It proposes an intelligent trade instruction recognition and execution pipeline, a noise-augmented 500-item dataset, and a comprehensive multi-metric evaluation across five LLMs. Key findings show strong generation and follow-up capabilities but substantial gaps in accuracy and completeness, underscoring risks in operational deployment. The proposed end-to-end pipeline demonstrates potential to improve reliability and security in automated trading, with practical implications for human-LLM collaboration in finance.

Abstract

The development of Large Language Models (LLMs) has created transformative opportunities for the financial industry, especially in the area of financial trading. However, how to integrate LLMs with trading systems has become a challenge. To address this problem, we propose an intelligent trade order recognition pipeline that enables the conversion of trade orders into a standard format in trade execution. The system improves the ability of human traders to interact with trading platforms while addressing the problem of misinformation acquisition in trade execution. In addition, we have created a trade order dataset of 500 pieces of data to simulate real-world trading scenarios. Moreover, we designed several metrics to provide a comprehensive assessment of dataset reliability and the generative power of big models in finance by experimenting with five state-of-the-art LLMs on our dataset. The results indicate that while LLMs demonstrate high generation rates (87.50% to 98.33%) and perfect follow-up rates, they face significant challenges in accuracy (5% to 10%) and completeness, with high missing rates (14.29% to 67.29%). In addition, LLMs tend to over-interrogate, suggesting that large models tend to collect more information, carrying certain challenges for information security.

Can Large Language Models Effectively Process and Execute Financial Trading Instructions?

TL;DR

This work tackles the challenge of translating natural-language financial trading instructions into executable commands. It proposes an intelligent trade instruction recognition and execution pipeline, a noise-augmented 500-item dataset, and a comprehensive multi-metric evaluation across five LLMs. Key findings show strong generation and follow-up capabilities but substantial gaps in accuracy and completeness, underscoring risks in operational deployment. The proposed end-to-end pipeline demonstrates potential to improve reliability and security in automated trading, with practical implications for human-LLM collaboration in finance.

Abstract

The development of Large Language Models (LLMs) has created transformative opportunities for the financial industry, especially in the area of financial trading. However, how to integrate LLMs with trading systems has become a challenge. To address this problem, we propose an intelligent trade order recognition pipeline that enables the conversion of trade orders into a standard format in trade execution. The system improves the ability of human traders to interact with trading platforms while addressing the problem of misinformation acquisition in trade execution. In addition, we have created a trade order dataset of 500 pieces of data to simulate real-world trading scenarios. Moreover, we designed several metrics to provide a comprehensive assessment of dataset reliability and the generative power of big models in finance by experimenting with five state-of-the-art LLMs on our dataset. The results indicate that while LLMs demonstrate high generation rates (87.50% to 98.33%) and perfect follow-up rates, they face significant challenges in accuracy (5% to 10%) and completeness, with high missing rates (14.29% to 67.29%). In addition, LLMs tend to over-interrogate, suggesting that large models tend to collect more information, carrying certain challenges for information security.

Paper Structure

This paper contains 20 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Dataset
  4. Introduction
  5. Preprocessing
  6. Data Alignment and Trading Instruction Classification
  7. Methodology
  8. Model
  9. Metric
  10. Experiment
  11. Experiment Setup
  12. Format Generation Evaluation
  13. Inquiry Capability Evaluation
  14. Result
  15. Format Generation Evaluation
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Format Generation Performance Across Models
  • Figure 2: Inquiry Capability Performance Across Models
  • Figure 3: Execution Pipeline
  • Figure 4: User Interaction