Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Is Temporal Difference Learning the Gold Standard for Stitching in RL?

Michał Bortkiewicz, Władysław Pałucki, Mateusz Ostaszewski, Benjamin Eysenbach

TL;DR

We address how agents stitch past experiences to solve long-horizon goal-conditioned tasks under function approximation. By designing a controlled grid-world benchmark with exact and generalized stitching, the study compares TD and MC methods and investigates architecture-scale effects. The main findings show MC can stitch in challenging settings, TD offers limited advantage in exact stitching but its benefit diminishes as task complexity grows, and increasing critic capacity drastically narrows generalization gaps for both paradigms, implying that scaling can supersede the traditional TD bias. The work suggests that effective stitching can emerge from model scale and representation, rather than specialized TD mechanisms, with implications for RL practice in large-model regimes.

Abstract

Reinforcement learning (RL) promises to solve long-horizon tasks even when training data contains only short fragments of the behaviors. This experience stitching capability is often viewed as the purview of temporal difference (TD) methods. However, outside of small tabular settings, trajectories never intersect, calling into question this conventional wisdom. Moreover, the common belief is that Monte Carlo (MC) methods should not be able to recombine experience, yet it remains unclear whether function approximation could result in a form of implicit stitching. The goal of this paper is to empirically study whether the conventional wisdom about stitching actually holds in settings where function approximation is used. We empirically demonstrate that Monte Carlo (MC) methods can also achieve experience stitching. While TD methods do achieve slightly stronger capabilities than MC methods (in line with conventional wisdom), that gap is significantly smaller than the gap between small and large neural networks (even on quite simple tasks). We find that increasing critic capacity effectively reduces the generalization gap for both the MC and TD methods. These results suggest that the traditional TD inductive bias for stitching may be less necessary in the era of large models for RL and, in some cases, may offer diminishing returns. Additionally, our results suggest that stitching, a form of generalization unique to the RL setting, might be achieved not through specialized algorithms (temporal difference learning) but rather through the same recipe that has provided generalization in other machine learning settings (via scale). Project website: https://michalbortkiewicz.github.io/golden-standard/

Is Temporal Difference Learning the Gold Standard for Stitching in RL?

TL;DR

We address how agents stitch past experiences to solve long-horizon goal-conditioned tasks under function approximation. By designing a controlled grid-world benchmark with exact and generalized stitching, the study compares TD and MC methods and investigates architecture-scale effects. The main findings show MC can stitch in challenging settings, TD offers limited advantage in exact stitching but its benefit diminishes as task complexity grows, and increasing critic capacity drastically narrows generalization gaps for both paradigms, implying that scaling can supersede the traditional TD bias. The work suggests that effective stitching can emerge from model scale and representation, rather than specialized TD mechanisms, with implications for RL practice in large-model regimes.

Abstract

Reinforcement learning (RL) promises to solve long-horizon tasks even when training data contains only short fragments of the behaviors. This experience stitching capability is often viewed as the purview of temporal difference (TD) methods. However, outside of small tabular settings, trajectories never intersect, calling into question this conventional wisdom. Moreover, the common belief is that Monte Carlo (MC) methods should not be able to recombine experience, yet it remains unclear whether function approximation could result in a form of implicit stitching. The goal of this paper is to empirically study whether the conventional wisdom about stitching actually holds in settings where function approximation is used. We empirically demonstrate that Monte Carlo (MC) methods can also achieve experience stitching. While TD methods do achieve slightly stronger capabilities than MC methods (in line with conventional wisdom), that gap is significantly smaller than the gap between small and large neural networks (even on quite simple tasks). We find that increasing critic capacity effectively reduces the generalization gap for both the MC and TD methods. These results suggest that the traditional TD inductive bias for stitching may be less necessary in the era of large models for RL and, in some cases, may offer diminishing returns. Additionally, our results suggest that stitching, a form of generalization unique to the RL setting, might be achieved not through specialized algorithms (temporal difference learning) but rather through the same recipe that has provided generalization in other machine learning settings (via scale). Project website: https://michalbortkiewicz.github.io/golden-standard/
Paper Structure (20 sections, 10 figures, 1 table)

This paper contains 20 sections, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. From tabular prediction to stitching.
  4. Preliminaries
  5. A Benchmark for Stitching
  6. Observation and action spaces.
  7. Interpreting setups difficulty: closed vs. open evaluation.
  8. Experiments
  9. Experimental setup
  10. Do any of today's methods do stitching?
  11. Are MC methods performing stitching, or is TD learning necessary?
  12. Does scaling improve stitching?
  13. Do quasimetric networks improve stitching?
  14. Conclusion
  15. Limitations.
  16. ...and 5 more sections

Figures (10)

  • Figure 1: (Left) While TD methods are often conceptualized as piecing together overlapping trajectories, (Right) this mental model breaks down in almost all realistic tasks, as trajectories never actually intersect. This paper introduces a new mental model (and formal definitions) for thinking about "stitching" in such settings, provides a benchmark for rigorously evaluating these stitching capabilities, and performs experiments to understand the degree to which stitching may actually achieved through (i) temporal difference methods, (ii) quasimetric architectures, and (iii) simply scaling model architectures.
  • Figure 2: Three types of stitching.
  • Figure 3: A benchmark for stitching: The agent (red ball) must move boxes to the target positions (yellow transparent boxes). (Left) During training, boxes are placed in one quarter and must be moved to an adjacent quarter (gray arrow indicates the required direction of transfer). During testing, boxes must be moved to the diagonal quarter. The gray arrows illustrate one of the valid two-step routes via adjacent quarters (adjacent $\rightarrow$ adjacent), which were seen separately during training but never as an end-to-end diagonal move. (Right) During training, one box is already on a target, and the agent must place the remaining two. During testing, no boxes start on targets. Although both start and goal configurations are individually familiar, training never includes segments that involve moving three boxes.
  • Figure 4: TD methods can only stitch effectively up to a certain point. In the Quarters setting (6×6 grid) — which tests exact stitching — increasing the number of boxes widens the generalization gap for both TD and MC methods.
  • Figure 5: A subtle failure of stitching. An agent trained on the quarters task (Fig. \ref{['fig:quarters']}) should first move all boxes to an adjacent corner and then to the goal quarter. However, if the agent prematurely moves a box along the diagonal, it will end up in a state that has never been seen before during training.
  • ...and 5 more figures