Lane2Seq: Towards Unified Lane Detection via Sequence Generation

TL;DR

Lane2Seq reframes lane detection as a sequence generation task to unify segmentation-, anchor-, and parameter-based formats within a single transformer-based encoder–decoder. By representing all outputs as token sequences under a shared vocabulary and using format prompts, it eliminates task-specific heads and losses. A reinforcement-learning-based multi-format tuning (MFRL) injects task-specific knowledge through format-specific rewards, improving performance across datasets without altering the model architecture. Empirically, Lane2Seq achieves state-of-the-art results on LLAMAS and Tusimple and competitive results on CULane, demonstrating the practicality of a simple seq2seq approach for unified lane detection. Limitations include slower inference for images with many lanes and the current scope of applying MFRL to seq2seq models, with ongoing work to speed up decoding and extend MFRL to other vision tasks.

Abstract

In this paper, we present a novel sequence generation-based framework for lane detection, called Lane2Seq. It unifies various lane detection formats by casting lane detection as a sequence generation task. This is different from previous lane detection methods, which depend on well-designed task-specific head networks and corresponding loss functions. Lane2Seq only adopts a plain transformer-based encoder-decoder architecture with a simple cross-entropy loss. Additionally, we propose a new multi-format model tuning based on reinforcement learning to incorporate the task-specific knowledge into Lane2Seq. Experimental results demonstrate that such a simple sequence generation paradigm not only unifies lane detection but also achieves competitive performance on benchmarks. For example, Lane2Seq gets 97.95\% and 97.42\% F1 score on Tusimple and LLAMAS datasets, establishing a new state-of-the-art result for two benchmarks.

Figures (6)

• Figure 1: Comparison of different lane detection frameworks.
• Figure 2: Inference pipeline of Lane2Seq. The model perceives input image, prompt and generates format-specific tokens, which can be detokenized into required detection format for visualization.
• Figure 3: Training pipeline of Lane2Seq. The input sequence can be the segmentation sequence or anchor sequence or parameter sequence.
• Figure 4: Visualization results of LaneATT, CLRNet, and Lane2Seq on Tusimple, CULane, and LLAMAS.
• Figure 5: Illustration of REINFORCE algorithm.