Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

See Less, Drive Better: Generalizable End-to-End Autonomous Driving via Foundation Models Stochastic Patch Selection

Amir Mallak, Erfan Aasi, Shiva Sreeram, Tsun-Hsuan Wang, Daniela Rus, Alaa Maalouf

TL;DR

The paper addresses redundancy in patch‑level features extracted from foundation models for end‑to‑end autonomous driving. It introduces Stochastic Patch Selection (SPS), which randomizes the subset of patch descriptors fed to the policy while preserving spatial layout, reducing compute by about $2.4\times$ and improving OOD performance by an average of $6.2\%$ (up to $20.4\%$ in hard cases). Empirical analyses show patch features lie in a low‑rank subspace with pervasive cross‑patch correlations; SPS exploits this by training on diverse, coherent views that emphasize invariant cues. The approach transfers from simulation to a real vehicle without tuning and can be paired with latent space text augmentation to further boost robustness, offering a practical, architecture‑agnostic pathway to robust FM‑based driving systems.

Abstract

Recent advances in end-to-end autonomous driving show that policies trained on patch-aligned features extracted from foundation models generalize better to Out-of-Distribution (OOD). We hypothesize that due to the self-attention mechanism, each patch feature implicitly embeds/contains information from all other patches, represented in a different way and intensity, making these descriptors highly redundant. We quantify redundancy in such (BLIP2) features via PCA and cross-patch similarity: $90$% of variance is captured by $17/64$ principal components, and strong inter-token correlations are pervasive. Training on such overlapping information leads the policy to overfit spurious correlations, hurting OOD robustness. We present Stochastic-Patch-Selection (SPS), a simple yet effective approach for learning policies that are more robust, generalizable, and efficient. For every frame, SPS randomly masks a fraction of patch descriptors, not feeding them to the policy model, while preserving the spatial layout of the remaining patches. Thus, the policy is provided with different stochastic but complete views of the (same) scene: every random subset of patches acts like a different, yet still sensible, coherent projection of the world. The policy thus bases its decisions on features that are invariant to which specific tokens survive. Extensive experiments confirm that across all OOD scenarios, our method outperforms the state of the art (SOTA), achieving a $6.2$% average improvement and up to $20.4$% in closed-loop simulations, while being $2.4\times$ faster. We conduct ablations over masking rates and patch-feature reorganization, training and evaluating 9 systems, with 8 of them surpassing prior SOTA. Finally, we show that the same learned policy transfers to a physical, real-world car without any tuning.

See Less, Drive Better: Generalizable End-to-End Autonomous Driving via Foundation Models Stochastic Patch Selection

TL;DR

The paper addresses redundancy in patch‑level features extracted from foundation models for end‑to‑end autonomous driving. It introduces Stochastic Patch Selection (SPS), which randomizes the subset of patch descriptors fed to the policy while preserving spatial layout, reducing compute by about and improving OOD performance by an average of (up to in hard cases). Empirical analyses show patch features lie in a low‑rank subspace with pervasive cross‑patch correlations; SPS exploits this by training on diverse, coherent views that emphasize invariant cues. The approach transfers from simulation to a real vehicle without tuning and can be paired with latent space text augmentation to further boost robustness, offering a practical, architecture‑agnostic pathway to robust FM‑based driving systems.

Abstract

Recent advances in end-to-end autonomous driving show that policies trained on patch-aligned features extracted from foundation models generalize better to Out-of-Distribution (OOD). We hypothesize that due to the self-attention mechanism, each patch feature implicitly embeds/contains information from all other patches, represented in a different way and intensity, making these descriptors highly redundant. We quantify redundancy in such (BLIP2) features via PCA and cross-patch similarity: % of variance is captured by principal components, and strong inter-token correlations are pervasive. Training on such overlapping information leads the policy to overfit spurious correlations, hurting OOD robustness. We present Stochastic-Patch-Selection (SPS), a simple yet effective approach for learning policies that are more robust, generalizable, and efficient. For every frame, SPS randomly masks a fraction of patch descriptors, not feeding them to the policy model, while preserving the spatial layout of the remaining patches. Thus, the policy is provided with different stochastic but complete views of the (same) scene: every random subset of patches acts like a different, yet still sensible, coherent projection of the world. The policy thus bases its decisions on features that are invariant to which specific tokens survive. Extensive experiments confirm that across all OOD scenarios, our method outperforms the state of the art (SOTA), achieving a % average improvement and up to % in closed-loop simulations, while being faster. We conduct ablations over masking rates and patch-feature reorganization, training and evaluating 9 systems, with 8 of them surpassing prior SOTA. Finally, we show that the same learned policy transfers to a physical, real-world car without any tuning.
Paper Structure (51 sections, 1 theorem, 14 equations, 6 figures, 5 tables)

This paper contains 51 sections, 1 theorem, 14 equations, 6 figures, 5 tables.

Table of Contents

  1. Introduction and background
  2. From foundation models to driving polices.
  3. Patch-wise features are redundant.
  4. Our contributions.
  5. Related work
  6. End-to-end autonomous driving.
  7. Foundation models in robotics and vision.
  8. Positioning and novelty.
  9. Method
  10. Preliminaries
  11. End-to-end driving as feature-based control.
  12. Multimodal patch-wise feature extraction Wang2024Drive.
  13. Mask design.
  14. Redundancy as low-rank structure
  15. Stochastic patch selection (SPS)
  16. ...and 36 more sections

Key Result

Lemma 1

Let $F_c \in \mathbb{R}^{N\times d}$ be a centered data matrix with $\operatorname{rank}(F_c)=r$, and let its (thin) singular value decomposition be $F_c \;=\; U_r \Sigma_r V_r^\top,$ where $U_r \in \mathbb{R}^{N\times r}$ and $V_r \in \mathbb{R}^{d\times r}$ have orthonormal columns. The orthogonal the following holds with probability at least $1-\delta$: i.e., the principal $r$-dimensional subs

Figures (6)

  • Figure 1: Stochastic Patch Selection (SPS) in a nutshell. (1) We use large vision-language models to extract patch-level features for input images. (2) These descriptors are often highly redundant and correlated. (3) SPS randomly masks a subset of descriptors, forcing the policy to learn based on different subsets and less correlation, improving efficiency ($\mathbf{2.4\times}$ speedup), generalization ($\mathbf{+6.2\%}$ performance), and enabling plug-and-play integration with downstream policies. In (4), we compare speed vs performance on different variants of SPS against SOTA.
  • Figure 2: SPS algorithm vs Drive-Anywhere illustration. From left to Right. 1st: Input images are processed through a frozen foundation model to produce patch-level descriptors. 2nd: In Drive-Anywhere, the full tensor is forwarded unchanged to the policy. 3rd: In our approach, we introduce two patch selection strategies (uniform stochastic and matrix-based probability selection), followed by (4th) a restructuring phase: either masking unselected descriptors or removing them and adjusting positional encodings. Both versions preserve spatial semantics while significantly reducing runtime. SPS improves efficiency by a factor of $\mathbf{2.4\times}$ while also boosting generalization.
  • Figure 3: (a) Cumulative explained variance over principal components for all patches versus the top-$128$ patches selected by $\ell_2$ norm. The red line marks $90\%$. Vertical markers indicate $17$ and $14$ components for all and top-$128$, respectively. (b) Patch-wise Pearson correlation matrix for one scenario. Strong off-diagonal correlation indicates widespread cross-patch redundancy. (c) Cosine-similarity overlays projected onto the image plane. Bright regions indicate patches whose descriptors are highly similar to the seed, visualizing global entanglement from self-attention.
  • Figure 4: Left: Diverse OOD sample frames across varying seasons, weather, and lighting conditions. Right: Real-car deployment representative frames from the rural road (public park) and parking garage, captured from both the onboard camera and external view.
  • Figure 5: Accuracy as a function of selection rate across variants.
  • ...and 1 more figures

Theorems & Definitions (2)

  • Lemma 1: SPS preserves the row-space under low rank and bounded coherence
  • proof : Proof of Lemma \ref{['lemma:1']}