Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Decision Making with Generative Action Sets

Jianyu Xu, Vidhi Jain, Bryan Wilder, Aarti Singh

TL;DR

This paper addresses online decision making when actions can be created on demand at a fixed cost and then reused. It introduces a doubly-optimistic algorithm that uses Lower Confidence Bounds for selecting among existing actions and Upper Confidence Bounds to decide when to generate new ones, balancing exploitation, exploration, and creation. The authors prove a sublinear regret of $O\Big(T^{\frac{d}{d+2}} d^{\frac{d}{d+2}} + d\sqrt{T\log T}\Big)$ with a matching $\Omega\big(T^{\frac{d}{d+2}}\big)$ lower bound, and validate the approach on healthcare Q&A tasks showing favorable generation-quality tradeoffs. The work advances online learning with expanding action spaces, offering practical impact for systems that must dynamically grow their knowledge bases while maintaining performance and cost efficiency.

Abstract

With advances in generative AI, decision-making agents can now dynamically create new actions during online learning, but action generation typically incurs costs that must be balanced against potential benefits. We study an online learning problem where an agent can generate new actions at any time step by paying a one-time cost, with these actions becoming permanently available for future use. The challenge lies in learning the optimal sequence of two-fold decisions: which action to take and when to generate new ones, further complicated by the triangular tradeoffs among exploitation, exploration and $\textit{creation}$. To solve this problem, we propose a doubly-optimistic algorithm that employs Lower Confidence Bounds (LCB) for action selection and Upper Confidence Bounds (UCB) for action generation. Empirical evaluation on healthcare question-answering datasets demonstrates that our approach achieves favorable generation-quality tradeoffs compared to baseline strategies. From theoretical perspectives, we prove that our algorithm achieves the optimal regret of $O(T^{\frac{d}{d+2}}d^{\frac{d}{d+2}} + d\sqrt{T\log T})$, providing the first sublinear regret bound for online learning with expanding action spaces.

Online Decision Making with Generative Action Sets

TL;DR

This paper addresses online decision making when actions can be created on demand at a fixed cost and then reused. It introduces a doubly-optimistic algorithm that uses Lower Confidence Bounds for selecting among existing actions and Upper Confidence Bounds to decide when to generate new ones, balancing exploitation, exploration, and creation. The authors prove a sublinear regret of with a matching lower bound, and validate the approach on healthcare Q&A tasks showing favorable generation-quality tradeoffs. The work advances online learning with expanding action spaces, offering practical impact for systems that must dynamically grow their knowledge bases while maintaining performance and cost efficiency.

Abstract

With advances in generative AI, decision-making agents can now dynamically create new actions during online learning, but action generation typically incurs costs that must be balanced against potential benefits. We study an online learning problem where an agent can generate new actions at any time step by paying a one-time cost, with these actions becoming permanently available for future use. The challenge lies in learning the optimal sequence of two-fold decisions: which action to take and when to generate new ones, further complicated by the triangular tradeoffs among exploitation, exploration and . To solve this problem, we propose a doubly-optimistic algorithm that employs Lower Confidence Bounds (LCB) for action selection and Upper Confidence Bounds (UCB) for action generation. Empirical evaluation on healthcare question-answering datasets demonstrates that our approach achieves favorable generation-quality tradeoffs compared to baseline strategies. From theoretical perspectives, we prove that our algorithm achieves the optimal regret of , providing the first sublinear regret bound for online learning with expanding action spaces.

Paper Structure

This paper contains 35 sections, 12 theorems, 32 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation (preview).
  3. Summary of Contributions
  4. Technical Novelty.
  5. Paper Organization.
  6. Related Works
  7. Problem Setup
  8. Algorithm
  9. Computational Complexity
  10. Theoretical Analysis
  11. Definitions and Assumptions
  12. Regret Bounds
  13. Empirical Performance
  14. Regret Validation on Synthetic Data
  15. Generation-Quality Tradeoffs Analysis on Healthcare Q&A Datasets
  16. ...and 20 more sections

Key Result

Theorem 5.4

With assumptions made in subsec:assumption, the expected regret of our algo:doubly_opt is upper bounded by $O(T^{\frac{d}{d+2}}d^{\frac{d}{d+2}} + d\sqrt{T\log T})$.

Figures (2)

  • Figure 1: Regret curves for $T=10000$ and $d=2,3,4$ in log-log scales, repeated by $N=10$ epochs. The slope of the linear asymptote under log-log diagram indicates the power dependence of regret on $T$, which should be $\frac{d}{d+2}$.
  • Figure 2: Tradeoffs between normalized generation costs (x-axis) and normalized mismatching loss (y-axis) on two healthcare Q&A datasets. A lower/left curve indicates a better performance. Each blue point represents the (generation cost, mismatching loss) pair caused by a choice of $c$. In both cases, our algorithm outperforms the baseline that randomly generates custom responses with a variety of fixed probabilities $p$ (each gray point represents a choice of $p$).

Theorems & Definitions (29)

  • Example 1.1: Healthcare Question-Answering Systems
  • Example 1.2: Personalized Advertisement
  • Definition 5.1: Optimal and Regret
  • Theorem 5.4: Regret upper bound
  • proof : Proof Sketch
  • Theorem 5.5: Regret lower bound
  • Lemma B.1: $OPT_o$ upper bound
  • proof : Proof sketch
  • Lemma B.2: $\Delta_t$ as estimation error
  • Lemma B.3: Constant competitive ratio
  • ...and 19 more