Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Balancing Exploration and Exploitation in LLM using Soft RLLF for Enhanced Negation Understanding

Ha-Thanh Nguyen, Ken Satoh

TL;DR

This work leverages Reinforcement Learning from Logical Feedback (RLLF) to create an effective balance between exploration and exploitation in LLMs and demonstrates the effectiveness of balancing exploration and exploitation with RLLF in improving LLMs' negation capabilities.

Abstract

Finetuning approaches in NLP often focus on exploitation rather than exploration, which may lead to suboptimal models. Given the vast search space of natural language, this limited exploration can restrict their performance in complex, high-stakes domains, where accurate negation understanding and logical reasoning abilities are crucial. To address this issue, we leverage Reinforcement Learning from Logical Feedback (RLLF) to create an effective balance between exploration and exploitation in LLMs. Our approach employs an appropriate benchmark dataset for training and evaluation, highlighting the importance of exploration in enhancing negation understanding capabilities. We compare the performance of our RLLF-enhanced LLMs with baseline models trained without RLLF, demonstrating the value of this balanced approach. Furthermore, we showcase the potential of our method in legal AI applications by employing transfer learning and evaluating its impact on negation understanding. Our experimental results exhibit the effectiveness of balancing exploration and exploitation with RLLF in improving LLMs' negation capabilities. This has implications for the development of more accurate, reliable, and logically consistent language models in high-stakes domains.

Balancing Exploration and Exploitation in LLM using Soft RLLF for Enhanced Negation Understanding

TL;DR

This work leverages Reinforcement Learning from Logical Feedback (RLLF) to create an effective balance between exploration and exploitation in LLMs and demonstrates the effectiveness of balancing exploration and exploitation with RLLF in improving LLMs' negation capabilities.

Abstract

Finetuning approaches in NLP often focus on exploitation rather than exploration, which may lead to suboptimal models. Given the vast search space of natural language, this limited exploration can restrict their performance in complex, high-stakes domains, where accurate negation understanding and logical reasoning abilities are crucial. To address this issue, we leverage Reinforcement Learning from Logical Feedback (RLLF) to create an effective balance between exploration and exploitation in LLMs. Our approach employs an appropriate benchmark dataset for training and evaluation, highlighting the importance of exploration in enhancing negation understanding capabilities. We compare the performance of our RLLF-enhanced LLMs with baseline models trained without RLLF, demonstrating the value of this balanced approach. Furthermore, we showcase the potential of our method in legal AI applications by employing transfer learning and evaluating its impact on negation understanding. Our experimental results exhibit the effectiveness of balancing exploration and exploitation with RLLF in improving LLMs' negation capabilities. This has implications for the development of more accurate, reliable, and logically consistent language models in high-stakes domains.
Paper Structure (16 sections, 3 equations, 4 figures, 4 tables)

This paper contains 16 sections, 3 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methodology
  4. Training Reward Model
  5. Reinforcement Learning on Logical Feedback
  6. Transfer Learning and Model Evaluation
  7. Experiment
  8. Model Choice
  9. Datasets
  10. Ruletaker Dataset ruletaker2020:
  11. Jina Dataset gunther2023jina:
  12. xNot360 Dataset nguyen2023negation:
  13. Experimental Settings
  14. Experimental Results
  15. Discussion
  16. ...and 1 more sections

Figures (4)

  • Figure 1: RLLF is the idea of allowing feedback for reinforcement learning to come not only from the user but also from the accuracy in the chain of logical reasoning. nguyen2023enhancing
  • Figure 2: Performance chart of GPT models on the xNot360 dataset. The chart displays a sinusoidal-like pattern, highlighting the differences in performance among the models.nguyen2023negation
  • Figure 3: Overview of the Reinforcement Learning from Logical Feedback (RLLF) methodology used to improve LLM's negation understanding capabilities, highlighting the exploration-exploitation balance and key steps involved.
  • Figure 4: Confusion matrices of GPT-2-ZS, GPT-2-TT, and GPT-2-RLLF-TT predictions on xNot360.