Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Action Shapley: A Training Data Selection Metric for World Model in Reinforcement Learning

Rajat Ghosh, Debojyoti Dutta

TL;DR

This work introduces Action Shapley, an agnostic data-valuing metric designed to select high-impact training data for world-model training in model-based reinforcement learning. A randomized dynamic algorithm enables scalable approximation of data-point Shapley values, balancing accuracy with computational efficiency. Across five real-world, data-constrained case studies, Action Shapley identifies indispensable data points and guides a data-selection policy that outperforms ad-hoc choices, achieving substantial computational savings (up to and exceeding 80% over exhaustive Shapley computations) while maintaining or improving cumulative reward performance. The approach is compatible with various world-model and RL algorithms, and its principled data-selection framework promises improved data efficiency and robustness for real-world dynamic control applications.

Abstract

Numerous offline and model-based reinforcement learning systems incorporate world models to emulate the inherent environments. A world model is particularly important in scenarios where direct interactions with the real environment is costly, dangerous, or impractical. The efficacy and interpretability of such world models are notably contingent upon the quality of the underlying training data. In this context, we introduce Action Shapley as an agnostic metric for the judicious and unbiased selection of training data. To facilitate the computation of Action Shapley, we present a randomized dynamic algorithm specifically designed to mitigate the exponential complexity inherent in traditional Shapley value computations. Through empirical validation across five data-constrained real-world case studies, the algorithm demonstrates a computational efficiency improvement exceeding 80\% in comparison to conventional exponential time computations. Furthermore, our Action Shapley-based training data selection policy consistently outperforms ad-hoc training data selection.

Action Shapley: A Training Data Selection Metric for World Model in Reinforcement Learning

TL;DR

This work introduces Action Shapley, an agnostic data-valuing metric designed to select high-impact training data for world-model training in model-based reinforcement learning. A randomized dynamic algorithm enables scalable approximation of data-point Shapley values, balancing accuracy with computational efficiency. Across five real-world, data-constrained case studies, Action Shapley identifies indispensable data points and guides a data-selection policy that outperforms ad-hoc choices, achieving substantial computational savings (up to and exceeding 80% over exhaustive Shapley computations) while maintaining or improving cumulative reward performance. The approach is compatible with various world-model and RL algorithms, and its principled data-selection framework promises improved data efficiency and robustness for real-world dynamic control applications.

Abstract

Numerous offline and model-based reinforcement learning systems incorporate world models to emulate the inherent environments. A world model is particularly important in scenarios where direct interactions with the real environment is costly, dangerous, or impractical. The efficacy and interpretability of such world models are notably contingent upon the quality of the underlying training data. In this context, we introduce Action Shapley as an agnostic metric for the judicious and unbiased selection of training data. To facilitate the computation of Action Shapley, we present a randomized dynamic algorithm specifically designed to mitigate the exponential complexity inherent in traditional Shapley value computations. Through empirical validation across five data-constrained real-world case studies, the algorithm demonstrates a computational efficiency improvement exceeding 80\% in comparison to conventional exponential time computations. Furthermore, our Action Shapley-based training data selection policy consistently outperforms ad-hoc training data selection.
Paper Structure (10 sections, 4 theorems, 6 equations, 6 figures, 6 tables, 1 algorithm)

This paper contains 10 sections, 4 theorems, 6 equations, 6 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Action Shapley
  3. Data Collection and Implementation
  4. Results
  5. Action Shapley Values for Training Data
  6. Validation of Action Shapley Based Training Data Selection Policy
  7. Comparison to Baseline Results
  8. Related Work
  9. Conclusion
  10. Appendix

Key Result

Proposition 1

We leverage the cut-off cardinality to circumvent the default exponential-time complexity and introduce a randomized dynamic algorithm denoted as Algorithm alg:ASC in Appendix.

Figures (6)

  • Figure 1: The RL training loop. (a) the training data is selected for the world model development. (b) the collected training data is used to train (update) the world model by a self-supervised algorithm. (c) With the world model, the agent updates its policy by using an RL algorithm. (d) The agent policy is evaluated. If needed, the agent goes back to step (a) for further improvement.
  • Figure 2: This plot compares the cumulative reward scores for the best Action Shapley agent vs the worst Action Shapley agent for the VM right-sizing case study.
  • Figure 3: Validation of Action Shapley based selection policy for the load balancing case study. (a) Comparisons of cumulative rewards among the best Action Shapley agent, the worst Action Shapley agent, and the best of 4 random training action sets for 25 different episodes. (b) Fractions of agents based on 4 random training datasets with lower cumulative rewards than that of the best Action Shapley agent for 25 episodes.
  • Figure 4: Validation of Action Shapley based selection policy for database tuning. (a) Comparisons of cumulative rewards among the best Action Shapley agent, the worst Action Shapley agent, and the best of 4 random training action sets for 25 different episodes. (b) Fractions of agents based on 4 random training datasets with lower cumulative rewards than that of the best Action Shapley agent for 25 episodes.
  • Figure 5: Validation of Action Shapley based selection policy for K8s management. (a) Comparisons of cumulative rewards among the best Action Shapley agent, the worst Action Shapley agent, and the best of 30 random training action sets for 25 different episodes. (b) Fractions of agents based on 30 random training datasets with lower cumulative rewards than that of the best Action Shapley agent for 25 episodes.
  • ...and 1 more figures

Theorems & Definitions (9)

  • Definition 1
  • Remark 1
  • Definition 2
  • Proposition 1
  • Remark 2
  • Definition 3
  • Corollary 1
  • Proposition 2
  • Proposition 3