Accelerating Structured Chain-of-Thought in Autonomous Vehicles

TL;DR

This work tackles the real-time latency challenge of chain-of-thought reasoning in autonomous driving VLA models by introducing FastDriveCoT, a parallel decoding framework. It combines a template-driven CoT with a dependency DAG and dynamic programming to schedule independent fields for concurrent generation, while a single-sequence attention mask and KV-cache sharing ensure efficient GPU utilization. Empirical results show 3.1–4.1× speedups in CoT generation and up to 3× reductions in end-to-end latency across multiple base models, with maintained or improved meta-action and trajectory performance. The approach leverages AV-specific CoT structure, including enumeration/elaboration for multi-instance fields, and demonstrates applicability across autoregressive and transfusion-style architectures, offering a practical pathway to deploy structured CoT in real-time driving settings.

Abstract

Chain-of-Thought (CoT) reasoning enhances the decision-making capabilities of vision-language-action models in autonomous driving, but its autoregressive nature introduces significant inference latency, making it impractical for real-time applications. To address this, we introduce FastDriveCoT, a novel parallel decoding method that accelerates template-structured CoT. Our approach decomposes the reasoning process into a dependency graph of distinct sub-tasks, such as identifying critical objects and summarizing traffic rules, some of which can be generated in parallel. By generating multiple independent reasoning steps concurrently within a single forward pass, we significantly reduce the number of sequential computations. Experiments demonstrate a 3-4$\times$ speedup in CoT generation and a substantial reduction in end-to-end latency across various model architectures, all while preserving the original downstream task improvements brought by incorporating CoT reasoning.

Figures (5)

• Figure 1: Introducing a CoT template can help improve the accuracy of meta-action prediction and trajectory generation in AVs. Further structuring CoT with a template containing specific topic dependencies makes CoT decoding parallelizable.
• Figure 2: Parallel decoding of structured chain-of-thought (CoT) in a VLA. The CoT is decomposed into a template of fields with certain dependencies, represented by edges. All independent fields, whose dependencies have been satisfied, are decoded at the same time, such as lane, traffic sign, and critical objects as shown in the figure. Background shading within the LLM shows the updating of the KV cache.
• Figure 3: An example part of the dependency graph and the corresponding attention mask. Due to independence and thus possibility of parallelism, for example, A, B, D, and E cannot attend to each other. Other field pairs with dependence can have the latter attend to the former as usual. Specifically, the fixed tokens are pre-filled so they can be attended to by all subsequent tokens regardless of dependencies. Padding tokens cannot be attended to by other tokens.
• Figure 4: Additional analysis on the generation time of each data sample in the test set.
• Figure : Parallel CoT decoding using dependency graph