Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Trade-R1: Bridging Verifiable Rewards to Stochastic Environments via Process-Level Reasoning Verification

Rui Sun, Yifan Sun, Sheng Xu, Li Zhao, Jing Li, Daxin Jiang, Cheng Hua, Zuo Bai

TL;DR

Trade-R1 tackles reward hacking in finance by introducing verifiable, process-level supervision for RL in stochastic markets. It combines a retrieval-augmented triangular verification protocol with two semantic reward strategies, FSR and DSR, and employs a two-stage verification to manage long financial contexts. Theoretical analysis explains how coupling semantic validity with returns reduces variance and amplifies true signals, while experiments across CN and US markets show that DSR provides superior cross-market generalization and reasoning fidelity. The work offers a practical, explainable approach to robust financial decision-making with potential for improved transferability and reduced hallucinations in LLM-based trading agents.

Abstract

Reinforcement Learning (RL) has enabled Large Language Models (LLMs) to achieve remarkable reasoning in domains like mathematics and coding, where verifiable rewards provide clear signals. However, extending this paradigm to financial decision is challenged by the market's stochastic nature: rewards are verifiable but inherently noisy, causing standard RL to degenerate into reward hacking. To address this, we propose Trade-R1, a model training framework that bridges verifiable rewards to stochastic environments via process-level reasoning verification. Our key innovation is a verification method that transforms the problem of evaluating reasoning over lengthy financial documents into a structured Retrieval-Augmented Generation (RAG) task. We construct a triangular consistency metric, assessing pairwise alignment between retrieved evidence, reasoning chains, and decisions to serve as a validity filter for noisy market returns. We explore two reward integration strategies: Fixed-effect Semantic Reward (FSR) for stable alignment signals, and Dynamic-effect Semantic Reward (DSR) for coupled magnitude optimization. Experiments on different country asset selection demonstrate that our paradigm reduces reward hacking, with DSR achieving superior cross-market generalization while maintaining the highest reasoning consistency.

Trade-R1: Bridging Verifiable Rewards to Stochastic Environments via Process-Level Reasoning Verification

TL;DR

Trade-R1 tackles reward hacking in finance by introducing verifiable, process-level supervision for RL in stochastic markets. It combines a retrieval-augmented triangular verification protocol with two semantic reward strategies, FSR and DSR, and employs a two-stage verification to manage long financial contexts. Theoretical analysis explains how coupling semantic validity with returns reduces variance and amplifies true signals, while experiments across CN and US markets show that DSR provides superior cross-market generalization and reasoning fidelity. The work offers a practical, explainable approach to robust financial decision-making with potential for improved transferability and reduced hallucinations in LLM-based trading agents.

Abstract

Reinforcement Learning (RL) has enabled Large Language Models (LLMs) to achieve remarkable reasoning in domains like mathematics and coding, where verifiable rewards provide clear signals. However, extending this paradigm to financial decision is challenged by the market's stochastic nature: rewards are verifiable but inherently noisy, causing standard RL to degenerate into reward hacking. To address this, we propose Trade-R1, a model training framework that bridges verifiable rewards to stochastic environments via process-level reasoning verification. Our key innovation is a verification method that transforms the problem of evaluating reasoning over lengthy financial documents into a structured Retrieval-Augmented Generation (RAG) task. We construct a triangular consistency metric, assessing pairwise alignment between retrieved evidence, reasoning chains, and decisions to serve as a validity filter for noisy market returns. We explore two reward integration strategies: Fixed-effect Semantic Reward (FSR) for stable alignment signals, and Dynamic-effect Semantic Reward (DSR) for coupled magnitude optimization. Experiments on different country asset selection demonstrate that our paradigm reduces reward hacking, with DSR achieving superior cross-market generalization while maintaining the highest reasoning consistency.
Paper Structure (36 sections, 15 equations, 4 figures, 5 tables)

This paper contains 36 sections, 15 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning RL: From Verifiable Rewards to Process-Level Supervision
  4. Reward Hacking Under Noisy Objectives
  5. Financial Decision Making
  6. Methodology
  7. Dataset Construction
  8. Task Formulation
  9. Financial Settings
  10. Semantic Reward Strategy
  11. Fixed-effect Semantic Reward
  12. Dynamic-effect Semantic Reward
  13. Theoretical Analysis
  14. Problem Formulation: Decomposition of Noisy Returns
  15. Variance Analysis of DSR
  16. ...and 21 more sections

Figures (4)

  • Figure 1: Overview of the Financial LLM Agent Training Architecture Integrating Reasoning Verification and the Asymmetric Semantic Gating (ASG) Mechanism. The system evaluates not only the raw market return ($r$) resulting from the stock selection decision ($d_t$) but also strictly assesses the quality and factual consistency of the model's reasoning process ($c_t$).
  • Figure 2: Cumulative net asset value (NAV) curves of different reward strategies on the A-Share market test set (July--October 2025).
  • Figure 3: Cumulative net asset value (NAV) curves of different reward strategies on the US market test set (July--October 2025).
  • Figure 4: Necessity of Asymmetry. The penalty evasion phenomenon is evident in the symmetric strategy: similarity score drops from $0.711$ (Profit) to $0.578$ (Loss). DSR maintains robustness ($0.966$ in Loss).