TL-GRPO: Turn-Level RL for Reasoning-Guided Iterative Optimization

TL;DR

This work tackles the challenge of iterative optimization with a fixed environment and per-turn rewards by treating it as a single-state POMDP and introducing Turn-Level GRPO (TL-GRPO). TL-GRPO uses turn-level group sampling and a unified verifiable turn reward to enable fine-grained credit assignment without increasing sampling costs, addressing limitations of trajectory-level GRPO and black-box methods. Through experiments on analog circuit sizing, TL-GRPO outperforms Bayesian optimization and standard GRPO, and a 30B model trained with TL-GRPO achieves state-of-the-art results within the same simulation budget, showing strong generalization to unseen circuit specifications. The approach has practical significance for reasoning-guided optimization in electronic design automation and broader tool-integrated RL applications for LLMs.

Abstract

Large language models have demonstrated strong reasoning capabilities in complex tasks through tool integration, which is typically framed as a Markov Decision Process and optimized with trajectory-level RL algorithms such as GRPO. However, a common class of reasoning tasks, iterative optimization, presents distinct challenges: the agent interacts with the same underlying environment state across turns, and the value of a trajectory is determined by the best turn-level reward rather than cumulative returns. Existing GRPO-based methods cannot perform fine-grained, turn-level optimization in such settings, while black-box optimization methods discard prior knowledge and reasoning capabilities. To address this gap, we propose Turn-Level GRPO (TL-GRPO), a lightweight RL algorithm that performs turn-level group sampling for fine-grained optimization. We evaluate TL-GRPO on analog circuit sizing (ACS), a challenging scientific optimization task requiring multiple simulations and domain expertise. Results show that TL-GRPO outperforms standard GRPO and Bayesian optimization methods across various specifications. Furthermore, our 30B model trained with TL-GRPO achieves state-of-the-art performance on ACS tasks under same simulation budget, demonstrating both strong generalization and practical utility.

Figures (11)

• Figure 1: Illustration of an iterative optimization task (demonstrated with analog circuit sizing). Given a netlist with a set of targets, the agent iteratively proposes new designs (actions) and receives observations of the fixed environment (simulation script). In this process, each turn yields an independent reward score from a unified reward function, and the value of the entire trajectory is determined by the best design achieving the highest reward among all turns.
• Figure 2: Overview of TL-GRPO. For an input query with initial observations, the LLM agent asynchronously performs turn-level group sampling. Each sampled action $a ^{(i)}_t \in \mathcal{A}_{\text{text}} \cup \mathcal{A}_{\text{tool\_call}}$ receives an observation with a turn-level reward, and a group-relative advantage is subsequently estimated. All turn-level rollouts are then collected into a single batch for policy gradient updates.
• Figure 3: Comparison of training dynamics for the three RL algorithms: Trajectory-Level GRPO, Single-Turn GRPO, and Turn-Level GRPO. (a) Training reward score over steps. (b) Actor model generation entropy. (c) Average turn response length.
• Figure 4: Comparison of turn-level performance for models trained with three RL algorithms on out-of-domain tasks. The turn-0 score represents the performance of initial values. Each line shows the average across all evaluation data.
• Figure 5: Example tool schema for analog circuit simulation.