Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PoLi-RL: A Point-to-List Reinforcement Learning Framework for Conditional Semantic Textual Similarity

Zixin Song, Bowen Zhang, Qian-Wen Zhang, Di Yin, Xing Sun, Chunping Li

TL;DR

This work targets Conditional Semantic Textual Similarity (C-STS), a task that requires nuanced similarity judgments under explicit conditions. It introduces PoLi-RL, a two-stage reinforcement learning framework that first grounds a cross-encoder in basic scoring with pointwise cues, then refines semantic discrimination using a hybrid reward that includes pointwise, pairwise, and listwise components. A key contribution is the Parallel Slice Ranking Reward (PSRR), which partitions completions into parallel slices to enable granular credit assignment for each candidate, improving learning signal fidelity. On the official C-STS benchmark, PoLi-RL establishes a new state-of-the-art with a Spearman correlation of 48.18, outperforming large closed- and open-source models, and demonstrates robustness and interpretability through extensive ablations and qualitative analyses. The findings suggest a powerful, generally applicable paradigm for training LLMs on complex ranking-based conditional tasks, with potential impact on retrieval, ranking, and conditional judgment systems.

Abstract

Conditional Semantic Textual Similarity (C-STS) measures the semantic proximity between text segments under a specific condition, thereby overcoming the ambiguity inherent in traditional STS. However, existing methods are largely confined to discriminative models, failing to fully integrate recent breakthroughs in the NLP community concerning Large Language Models (LLMs) and Reinforcement Learning (RL). RL is a particularly well-suited paradigm for this task, as it can directly optimize the non-differentiable Spearman ranking metric and guide the reasoning process required by C-STS. However, we find that naively applying listwise RL fails to produce meaningful improvements, as the model is overwhelmed by complex, coarse-grained reward signals. To address this challenge, we introduce PoLi-RL, a novel Point-to-List Reinforcement Learning framework. PoLi-RL employs a two-stage curriculum: it first trains the model with simple pointwise rewards to establish fundamental scoring capabilities, then transitions to a hybrid reward that combines pointwise, pairwise, and listwise objectives to refine the model's ability to discern subtle semantic distinctions. Crucially, we propose an innovative Parallel Slice Ranking Reward (PSRR) mechanism that computes ranking rewards in parallel slices, where each slice comprises same-indexed completions from different samples. This provides a precise, differentiated learning signal for each individual completion, enabling granular credit assignment and effective optimization. On the official C-STS benchmark, PoLi-RL achieves a Spearman correlation coefficient of 48.18, establishing a new SOTA for the cross-encoder architecture. As the first work to successfully apply RL to C-STS, our study introduces a powerful and precise paradigm for training LLMs on complex, ranking-based conditional judgment tasks.

PoLi-RL: A Point-to-List Reinforcement Learning Framework for Conditional Semantic Textual Similarity

TL;DR

This work targets Conditional Semantic Textual Similarity (C-STS), a task that requires nuanced similarity judgments under explicit conditions. It introduces PoLi-RL, a two-stage reinforcement learning framework that first grounds a cross-encoder in basic scoring with pointwise cues, then refines semantic discrimination using a hybrid reward that includes pointwise, pairwise, and listwise components. A key contribution is the Parallel Slice Ranking Reward (PSRR), which partitions completions into parallel slices to enable granular credit assignment for each candidate, improving learning signal fidelity. On the official C-STS benchmark, PoLi-RL establishes a new state-of-the-art with a Spearman correlation of 48.18, outperforming large closed- and open-source models, and demonstrates robustness and interpretability through extensive ablations and qualitative analyses. The findings suggest a powerful, generally applicable paradigm for training LLMs on complex ranking-based conditional tasks, with potential impact on retrieval, ranking, and conditional judgment systems.

Abstract

Conditional Semantic Textual Similarity (C-STS) measures the semantic proximity between text segments under a specific condition, thereby overcoming the ambiguity inherent in traditional STS. However, existing methods are largely confined to discriminative models, failing to fully integrate recent breakthroughs in the NLP community concerning Large Language Models (LLMs) and Reinforcement Learning (RL). RL is a particularly well-suited paradigm for this task, as it can directly optimize the non-differentiable Spearman ranking metric and guide the reasoning process required by C-STS. However, we find that naively applying listwise RL fails to produce meaningful improvements, as the model is overwhelmed by complex, coarse-grained reward signals. To address this challenge, we introduce PoLi-RL, a novel Point-to-List Reinforcement Learning framework. PoLi-RL employs a two-stage curriculum: it first trains the model with simple pointwise rewards to establish fundamental scoring capabilities, then transitions to a hybrid reward that combines pointwise, pairwise, and listwise objectives to refine the model's ability to discern subtle semantic distinctions. Crucially, we propose an innovative Parallel Slice Ranking Reward (PSRR) mechanism that computes ranking rewards in parallel slices, where each slice comprises same-indexed completions from different samples. This provides a precise, differentiated learning signal for each individual completion, enabling granular credit assignment and effective optimization. On the official C-STS benchmark, PoLi-RL achieves a Spearman correlation coefficient of 48.18, establishing a new SOTA for the cross-encoder architecture. As the first work to successfully apply RL to C-STS, our study introduces a powerful and precise paradigm for training LLMs on complex, ranking-based conditional judgment tasks.

Paper Structure

This paper contains 27 sections, 9 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Formulation
  4. Reinforcement Learning for C-STS
  5. PoLi-RL: A Two-Stage Reinforcement Learning Framework
  6. Stage I: Foundational Skill Acquisition.
  7. Stage II: Fine-Grained Semantic Refinement.
  8. PSRR: A Fine-Grained Reward Mechanism.
  9. Pairwise Ranking Reward.
  10. Listwise Ranking Reward.
  11. Experiments
  12. Experimental Setup
  13. Main Results
  14. Ablation Studies
  15. Effectiveness of the Two-Stage Curriculum and Reward Components.
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Performance comparison of different strategies on the C-STS task. Directly applying listwise ranking rewards for RL does not significantly outperform the few-shot baseline. In contrast, both stages of our method (PoLi-RL) achieve substantial improvements, validating its effectiveness.
  • Figure 2: An overview of the PoLi-RL framework. The framework employs a two-stage curriculum, progressing from Stage I, where the model learns foundational scoring rules, to Stage II, which refines the model's ability to discern fine-grained semantic differences. The core of our method is the PSRR mechanism in Stage II, where pairwise and listwise ranking rewards are computed vertically within slices of same-indexed completions to provide precise, differentiated learning signals.
  • Figure 3: Distribution of absolute prediction errors for the raw model, SFT model, and PoLi-RL. The $x$-axis represents the absolute error ($\left|\text{Predicted}-\text{True}\right|$), and the $y$-axis is the probability density.
  • Figure 4: A detailed case study comparing the reasoning processes and final outputs of the Few-shot, SFT baselines, and PoLi-RL on a C-STS sample that requires nuanced reasoning.