Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generation

Do June Min, Veronica Perez-Rosas, Kenneth Resnicow, Rada Mihalcea

TL;DR

This paper introduces two novel bandit methods, DynaOpt and C-DynaOpt, which rely on the broad strategy of combining rewards into a single value and optimizing them simultaneously, and employs non-contextual and contextual multi-arm bandits to dynamically adjust multiple reward weights during training.

Abstract

In this paper, we study the problem of multi-reward reinforcement learning to jointly optimize for multiple text qualities for natural language generation. We focus on the task of counselor reflection generation, where we optimize the generators to simultaneously improve the fluency, coherence, and reflection quality of generated counselor responses. We introduce two novel bandit methods, DynaOpt and C-DynaOpt, which rely on the broad strategy of combining rewards into a single value and optimizing them simultaneously. Specifically, we employ non-contextual and contextual multi-arm bandits to dynamically adjust multiple reward weights during training. Through automatic and manual evaluations, we show that our proposed techniques, DynaOpt and C-DynaOpt, outperform existing naive and bandit baselines, showcasing their potential for enhancing language models.

Dynamic Reward Adjustment in Multi-Reward Reinforcement Learning for Counselor Reflection Generation

TL;DR

This paper introduces two novel bandit methods, DynaOpt and C-DynaOpt, which rely on the broad strategy of combining rewards into a single value and optimizing them simultaneously, and employs non-contextual and contextual multi-arm bandits to dynamically adjust multiple reward weights during training.

Abstract

In this paper, we study the problem of multi-reward reinforcement learning to jointly optimize for multiple text qualities for natural language generation. We focus on the task of counselor reflection generation, where we optimize the generators to simultaneously improve the fluency, coherence, and reflection quality of generated counselor responses. We introduce two novel bandit methods, DynaOpt and C-DynaOpt, which rely on the broad strategy of combining rewards into a single value and optimizing them simultaneously. Specifically, we employ non-contextual and contextual multi-arm bandits to dynamically adjust multiple reward weights during training. Through automatic and manual evaluations, we show that our proposed techniques, DynaOpt and C-DynaOpt, outperform existing naive and bandit baselines, showcasing their potential for enhancing language models.
Paper Structure (41 sections, 6 equations, 5 figures, 6 tables, 1 algorithm)

This paper contains 41 sections, 6 equations, 5 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning.
  4. Multi-reward Optimization.
  5. Multi-armed Bandits.
  6. NLP and Psychotherapy.
  7. DynaOpt: Dynamically Adjusting Rewards for Multiple Rewards Reinforcement Learning
  8. Multi-reward Optimization with Multi-armed Bandits
  9. Choosing an Action.
  10. Updating Reward Weights.
  11. Bandit Reward Computation.
  12. Updating the Bandit.
  13. C-DynaOpt: Reward Update with Contextual Multi-armed Bandits.
  14. Datasets and Task
  15. Datasets
  16. ...and 26 more sections

Figures (5)

  • Figure 1: Workflow of RL training of language models with multiple rewards. The main question is how to simultaneously use the multiple rewards to update the LM (Step 4).
  • Figure 2: An overview of the DynaOpt workflow. At each bandit step, the bandit pulls an arm, which updates the reward weights. Then, the RL optimization phase uses the summed weights and updates the language model. In the bandit update phase, the LM generations are scored by the reward models, and the scores are used to update the bandit model.
  • Figure 3: Sample reflection generations of different models on the counselor reflection generation task.
  • Figure 4: Reward weight trajectory of the DynaOpt model on the counselor reflection generation task.
  • Figure 5: Bandit arm weight history of the DynaOpt model on the counselor reflection generation task.