Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs for High-Frequency Decision-Making: Normalized Action Reward-Guided Consistency Policy Optimization

Yang Zhao, Zihao Li, Zhiyu Jiang, Dandan Ma, Ganchao Liu, Wenzhe Zhao

TL;DR

This paper proposes Normalized Action Reward guided Consistency Policy Optimization (NAR-CP), which first acquires predefined dense rewards from environmental feedback of candidate actions via reward functions, then completes reward shaping through normalization, and theoretically verifies action reward normalization does not impair optimal policy.

Abstract

While Large Language Models (LLMs) form the cornerstone of sequential decision-making agent development, they have inherent limitations in high-frequency decision tasks. Existing research mainly focuses on discrete embodied decision scenarios with low-frequency and significant semantic differences in state space (e.g., household planning). These methods suffer from limited performance in high-frequency decision-making tasks, since high-precision numerical state information in such tasks undergoes frequent updates with minimal fluctuations, and exhibiting policy misalignment between the learned sub-tasks and composite tasks. To address these issues, this paper proposes Normalized Action Reward guided Consistency Policy Optimization (NAR-CP). 1) Our method first acquires predefined dense rewards from environmental feedback of candidate actions via reward functions, then completes reward shaping through normalization, and theoretically verifies action reward normalization does not impair optimal policy. 2) To reduce policy misalignment in composite tasks, we use LLMs to infer sub-observation candidate actions and generate joint policies, with consistency loss ensuring precise alignment between global semantic policies and sub-semantic policies. Experiments on UAV pursuit, a typical high-frequency task, show our method delivers superior performance on independent and composite tasks with excellent generalization to unseen tasks.

LLMs for High-Frequency Decision-Making: Normalized Action Reward-Guided Consistency Policy Optimization

TL;DR

This paper proposes Normalized Action Reward guided Consistency Policy Optimization (NAR-CP), which first acquires predefined dense rewards from environmental feedback of candidate actions via reward functions, then completes reward shaping through normalization, and theoretically verifies action reward normalization does not impair optimal policy.

Abstract

While Large Language Models (LLMs) form the cornerstone of sequential decision-making agent development, they have inherent limitations in high-frequency decision tasks. Existing research mainly focuses on discrete embodied decision scenarios with low-frequency and significant semantic differences in state space (e.g., household planning). These methods suffer from limited performance in high-frequency decision-making tasks, since high-precision numerical state information in such tasks undergoes frequent updates with minimal fluctuations, and exhibiting policy misalignment between the learned sub-tasks and composite tasks. To address these issues, this paper proposes Normalized Action Reward guided Consistency Policy Optimization (NAR-CP). 1) Our method first acquires predefined dense rewards from environmental feedback of candidate actions via reward functions, then completes reward shaping through normalization, and theoretically verifies action reward normalization does not impair optimal policy. 2) To reduce policy misalignment in composite tasks, we use LLMs to infer sub-observation candidate actions and generate joint policies, with consistency loss ensuring precise alignment between global semantic policies and sub-semantic policies. Experiments on UAV pursuit, a typical high-frequency task, show our method delivers superior performance on independent and composite tasks with excellent generalization to unseen tasks.
Paper Structure (23 sections, 21 equations, 4 figures, 4 tables)

This paper contains 23 sections, 21 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Dense Reward shaping in RL.
  4. NAR-CP
  5. Dense Reward to Language-augmented RL
  6. Normalized Action Reward
  7. Normalize.
  8. Gradient Norm.
  9. Optimal Policy.
  10. Decoupled Consistency Policy
  11. Experiments
  12. Experimental Overview
  13. Experiment 1: Comparison Experiment
  14. Experimental Setup
  15. Results and Analysis
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Method Overview. (a) Reduce the deviation between the sub-tasks joint policy $\pi_J$ and the composite policy $\pi_F$ through the consistency loss. (b) By applying Action Reward Normalization to the original dense reward, we dynamically amplify the reward variance and restore the effectiveness of gradient signals.
  • Figure 2: Details Of The Consistency Policy. $\mathcal{A}_{\text{top-k}}$ represents the action with the highest probability extracted from the joint policy $\pi_J$ .$\pi_{\text{Full-topk}}$ obtained by inputting the combination of $\mathcal{A}_{\text{top-k}}$ and composite observation prompt into the LLM. Finally employ the KL divergence to construct the consistency loss $\mathcal{L}_{\text{consistency}}$
  • Figure 3: Figure from left to right in turn shows: The reward curve in the direction-tracking task. The reward curve in the distance-keeping task. The reward curve in the Integrated-tracking task.
  • Figure 4: Error distribution characteristics: Direction tracking: $x$-axis = root sum of squared deviations, $y$-axis = density; red zone = $0$--$0.141$ (precision), yellow zone = $0$--$0.707$ (general); accuracy increases as curve approaches 0 in positive zone. Distance keeping: $x$-axis = current $-$ target distance, $y$-axis = density; red zone = $\pm 0.1$ (precision), yellow zone = $\pm 1.0$ (general); accuracy increases as curve approaches 0. Integrated Tracking: $x$-axis = direction + distance error (+ = distance > target, $-$ = distance < target), $y$-axis = density; red zone = $\pm 0.241$ (precision), yellow zone = $\pm 1.707$ (general); coordinated control increases as curve approaches 0.