LTDA-Drive: LLMs-guided Generative Models based Long-tail Data Augmentation for Autonomous Driving

TL;DR

LTDA-Drive tackles long-tail distribution in 3D autonomous driving perception by a three-stage, LLM-guided data augmentation framework that replaces head-class cars with tail-class pedestrians and cyclists. It combines localized diffusion-based head-object removal, tail-object insertion, and LLM-driven candidate filtering to produce high-quality, diverse tail data. On KITTI, it delivers substantial gains for rare classes, notably up to 34% in 3D detection metrics, validating the approach's practicality for safety-critical perception. The work demonstrates a scalable path to reduce bias toward head classes and improve robustness for vulnerable road users, while acknowledging limitations in non-overlapping removal and video-frame consistency for future improvements.

Abstract

3D perception plays an essential role for improving the safety and performance of autonomous driving. Yet, existing models trained on real-world datasets, which naturally exhibit long-tail distributions, tend to underperform on rare and safety-critical, vulnerable classes, such as pedestrians and cyclists. Existing studies on reweighting and resampling techniques struggle with the scarcity and limited diversity within tail classes. To address these limitations, we introduce LTDA-Drive, a novel LLM-guided data augmentation framework designed to synthesize diverse, high-quality long-tail samples. LTDA-Drive replaces head-class objects in driving scenes with tail-class objects through a three-stage process: (1) text-guided diffusion models remove head-class objects, (2) generative models insert instances of the tail classes, and (3) an LLM agent filters out low-quality synthesized images. Experiments conducted on the KITTI dataset show that LTDA-Drive significantly improves tail-class detection, achieving 34.75\% improvement for rare classes over counterpart methods. These results further highlight the effectiveness of LTDA-Drive in tackling long-tail challenges by generating high-quality and diverse data.

Figures (6)

• Figure 1: Overview: LTDA-Drive augments data for long-tail classes by replacing frequent objects in driving scenes with rare objects through three key modules. 1) Head-class object removal: identifies removable objects from the original scenes and performs targeted latent diffusion inpainting based on image, mask, and text guidance to generate object-removed candidates. 2) Tail-class object insertion: receives object-removed scenes as input and identifies suitable locations for inserting tail-class objects, followed by localized inpainting to generate candidates. 3) LLM-guided candidate filtering: leverages an LLM agent to assess the quality, plausibility, and geometric alignment of the generated candidates, to preserve only high-quality generations.
• Figure 2: The LLM agent is presented with a pair of images, one containing a head-class object (car) and one after object removal. The agent is asked whether the object was removed cleanly. The response includes a binary yes/no decision and a quality score.
• Figure 3: The LLM agent is asked to evaluate the 3D bounding box alignment of the inserted tail-class object (cyclist) using in-context learning with positive and negative references. The response includes a binary yes/no decision followed by a brief explanation.
• Figure 4: The LLM agent is provided with a pair of images, one containing the original head-class object (car) and the other one containing the inserted tail-class object (cyclist). The agent is asked to identify if both objects are roughly showing the same side. The response includes a yes/no decision, followed by an explanation.
• Figure 5: Representative results from the LTDA-Drive augmentation process. The first column shows the original image before augmentation. The second column presents images with head-class objects removed. The third column displays the final augmented images, where tail-class objects are inserted. The first two rows show the augmentation results for pedestrians, while the last two rows show the results for cyclists. Red arrows highlight the original objects to be removed, the regions where removal occurs, and the inserted objects.