Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RLSF: Fine-tuning LLMs via Symbolic Feedback

Piyush Jha, Prithwish Jana, Pranavkrishna Suresh, Arnav Arora, Vijay Ganesh

TL;DR

RLSF addresses the gap where LLMs struggle with domain-specific reasoning by leveraging symbolic reasoning tools to provide fine-grained, token-level feedback via poly-sized certificates. The authors formalize Reinforcement Learning via Symbolic Feedback (RLSF), combining an LLM as the RL agent with an environment that uses symbolic tools to certify outputs and supply corrective feedback without requiring differentiable reasoning. They evaluate RLSF across five reasoning tasks: NL-to-C++ code, three chemistry tasks (MG, FS, RS), and Game of 24 with Tree of Thoughts, showing substantial gains over SFT and RL baselines and even outperforming much larger models in several settings. The results demonstrate that token-level symbolic feedback can dramatically improve domain-specific accuracy while enabling smaller LLMs to surpass much larger closed-source models, highlighting RLSF's practical potential and scalability.

Abstract

Large Language Models (LLMs) have transformed AI but often struggle with tasks that require domain-specific reasoning and logical alignment. Traditional fine-tuning methods do not leverage the vast amount of symbolic domain-knowledge available to us via symbolic reasoning tools (e.g., provers), and are further limited by sparse rewards and unreliable reward models. We introduce Reinforcement Learning via Symbolic Feedback (RLSF), a novel fine-tuning paradigm where symbolic reasoning tools (e.g., solvers, provers, and algebra systems) provide fine-grained feedback to LLMs. RLSF uses poly-sized certificates (e.g., proofs) generated by symbolic tools to identify and correct errors in model outputs, offering token-level guidance without requiring differentiable reasoning systems. This paradigm bridges the gap between symbolic reasoning and LLM fine-tuning, enabling precise alignment with domain-specific constraints while addressing key limitations of traditional reward signals. Via extensive evaluations, we show that our RLSF-based fine-tuning of LLMs outperforms traditional approaches on five different applications (that have some associated logical or domain constraints), namely, program synthesis from natural language pseudo-code to programming language, three chemistry tasks, and solving the Game of 24. A key takeaway is that fine-tuning via RLSF enables relatively smaller LLMs to significantly outperform closed-source models that are orders of magnitude larger.

RLSF: Fine-tuning LLMs via Symbolic Feedback

TL;DR

RLSF addresses the gap where LLMs struggle with domain-specific reasoning by leveraging symbolic reasoning tools to provide fine-grained, token-level feedback via poly-sized certificates. The authors formalize Reinforcement Learning via Symbolic Feedback (RLSF), combining an LLM as the RL agent with an environment that uses symbolic tools to certify outputs and supply corrective feedback without requiring differentiable reasoning. They evaluate RLSF across five reasoning tasks: NL-to-C++ code, three chemistry tasks (MG, FS, RS), and Game of 24 with Tree of Thoughts, showing substantial gains over SFT and RL baselines and even outperforming much larger models in several settings. The results demonstrate that token-level symbolic feedback can dramatically improve domain-specific accuracy while enabling smaller LLMs to surpass much larger closed-source models, highlighting RLSF's practical potential and scalability.

Abstract

Large Language Models (LLMs) have transformed AI but often struggle with tasks that require domain-specific reasoning and logical alignment. Traditional fine-tuning methods do not leverage the vast amount of symbolic domain-knowledge available to us via symbolic reasoning tools (e.g., provers), and are further limited by sparse rewards and unreliable reward models. We introduce Reinforcement Learning via Symbolic Feedback (RLSF), a novel fine-tuning paradigm where symbolic reasoning tools (e.g., solvers, provers, and algebra systems) provide fine-grained feedback to LLMs. RLSF uses poly-sized certificates (e.g., proofs) generated by symbolic tools to identify and correct errors in model outputs, offering token-level guidance without requiring differentiable reasoning systems. This paradigm bridges the gap between symbolic reasoning and LLM fine-tuning, enabling precise alignment with domain-specific constraints while addressing key limitations of traditional reward signals. Via extensive evaluations, we show that our RLSF-based fine-tuning of LLMs outperforms traditional approaches on five different applications (that have some associated logical or domain constraints), namely, program synthesis from natural language pseudo-code to programming language, three chemistry tasks, and solving the Game of 24. A key takeaway is that fine-tuning via RLSF enables relatively smaller LLMs to significantly outperform closed-source models that are orders of magnitude larger.
Paper Structure (26 sections, 4 figures, 4 tables, 1 algorithm)

This paper contains 26 sections, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Related Work
  2. Reinforcement Learning via Symbolic Feedback (RLSF)
  3. Reasoning Task A: Natural Language Pseudo-code to C++ Code
  4. Benchmark and RLSF setup
  5. Evaluation Metrics
  6. Comparative Models Used
  7. Results and Ablation Study
  8. Reasoning Tasks B: Chemistry (Molecule Generation, Forward Synthesis and Retrosynthesis)
  9. Benchmark and RLSF setup
  10. Evaluation metrics
  11. Comparative Models Used
  12. Results and Ablation Study
  13. Reasoning Task C: Game of 24 using Tree of Thoughts (ToT)
  14. Benchmark and RLSF setup
  15. Comparative Methods and Models Used
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Contrasting RLHF with RLSF: The image depicts two distinct fine-tuning paradigms. (Top) RLHF operates within an environment governed by a black-box reward model, typically offering scalar feedback. (Bottom) By contrast, the environment in RLSF leverages sound symbolic reasoning tools and also provides token-level feedback that is, in turn, based on poly-sized certificates produced by these symbolic tools.
  • Figure 2: RLSF for translation of NL pseudo-code to code: Given the generated C++ code (with N lines), the symbolic environment uses the g++ compiler to detect erroneous lines ($E$) and compute the pass rate $r$ from the test suite, providing fine-grained symbolic feedback for fine-tuning the LLM.
  • Figure 3: RLSF for one of the chemistry tasks - Molecule Generation: In this illustration, the symbolic environment utilizes RDKit to generate a token-level reward vector as feedback based on the presence or absence of any syntactical errors. Moreover, for the semantic errors, we again use RDKit to check for the presence of the required functional groups mentioned in the input natural language description and penalize the entire generated molecule if it lacks the required functional groups. Each element in the reward vector corresponds to a token in the response, where erroneous tokens are penalized with a value of 0 and correct ones are assigned 1. The last element of the reward vector (corresponding to the <EOS> token) is 1 only if the entire response is correct, otherwise, it is 0.
  • Figure 4: RLSF for the Game of 24: In this illustration, the symbolic environment utilizes the Computer Algebra System (CAS) library SymPy to generate a token-level reward vector as feedback. Each element in the vector corresponds to a token in the response, where erroneous tokens are penalized with a value of 0 and correct ones are assigned 1. The last element of the reward vector (corresponding to the EOS token) is 1 only if the entire response is correct, otherwise, it is 0.