Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Tokenization via Reinforcement Patching: End-to-end Training and Zero-shot Transfer

Yulun Wu, Sravan Kumar Ankireddy, Samuel Sharpe, Nikita Seleznev, Dehao Yuan, Hyeji Kim, Nam H. Nguyen

Abstract

Efficiently aggregating spatial or temporal horizons to acquire compact representations has become a unifying principle in modern deep learning models, yet learning data-adaptive representations for long-horizon sequence data, especially continuous sequences like time series, remains an open challenge. While fixed-size patching has improved scalability and performance, discovering variable-sized, data-driven patches end-to-end often forces models to rely on soft discretization, specific backbones, or heuristic rules. In this work, we propose Reinforcement Patching (ReinPatch), the first framework to jointly optimize a sequence patching policy and its downstream sequence backbone model using reinforcement learning. By formulating patch boundary placement as a discrete decision process optimized via Group Relative Policy Gradient (GRPG), ReinPatch bypasses the need for continuous relaxations and performs dynamic patching policy optimization in a natural manner. Moreover, our method allows strict enforcement of a desired compression rate, freeing the downstream backbone to scale efficiently, and naturally supports multi-level hierarchical modeling. We evaluate ReinPatch on time-series forecasting datasets, where it demonstrates compelling performance compared to state-of-the-art data-driven patching strategies. Furthermore, our detached design allows the patching module to be extracted as a standalone foundation patcher, providing the community with visual and empirical insights into the segmentation behaviors preferred by a purely performance-driven neural patching strategy.

Dynamic Tokenization via Reinforcement Patching: End-to-end Training and Zero-shot Transfer

Abstract

Efficiently aggregating spatial or temporal horizons to acquire compact representations has become a unifying principle in modern deep learning models, yet learning data-adaptive representations for long-horizon sequence data, especially continuous sequences like time series, remains an open challenge. While fixed-size patching has improved scalability and performance, discovering variable-sized, data-driven patches end-to-end often forces models to rely on soft discretization, specific backbones, or heuristic rules. In this work, we propose Reinforcement Patching (ReinPatch), the first framework to jointly optimize a sequence patching policy and its downstream sequence backbone model using reinforcement learning. By formulating patch boundary placement as a discrete decision process optimized via Group Relative Policy Gradient (GRPG), ReinPatch bypasses the need for continuous relaxations and performs dynamic patching policy optimization in a natural manner. Moreover, our method allows strict enforcement of a desired compression rate, freeing the downstream backbone to scale efficiently, and naturally supports multi-level hierarchical modeling. We evaluate ReinPatch on time-series forecasting datasets, where it demonstrates compelling performance compared to state-of-the-art data-driven patching strategies. Furthermore, our detached design allows the patching module to be extracted as a standalone foundation patcher, providing the community with visual and empirical insights into the segmentation behaviors preferred by a purely performance-driven neural patching strategy.

Paper Structure

This paper contains 23 sections, 9 equations, 3 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Modeling Framework
  5. Generative sequence patching as multi-step MDP
  6. Non-causal sequence patching as contextual bandit
  7. Unified one-step MDP for efficient optimization
  8. Objectives and Optimization
  9. How is desired compression rate enforced?
  10. Why is GRPO not preferred over GRPG?
  11. Backbone Architecture
  12. Extension to multi-level hierarchical model
  13. Inference Time Adaptation
  14. Experiments
  15. Experimental Details
  16. ...and 8 more sections

Figures (3)

  • Figure 1: MSE at forecasting horizon $H=96$ across various look-back windows. While entropy patching exhibits a competitive for short contexts, it struggles to scale as the input context expands, whereas ReinPatch maintains strong forecasting performance across most datasets and context lengths.
  • Figure 2: Representative patch segmentations sampled from the foundation patcher trained with a minimum compression rate of 8. The learned boundaries naturally concentrate around major transitions while keeping short neighborhoods of local temporal structure together.
  • Figure 3: Sequence backbone model. Note that this illustration is for contextual tasks. For generative (i.e. causal) tasks, patches are right-shifted by one (and a special token embedding is added to the beginning of the sequence) before upsampling.