Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are All Data Necessary? Efficient Data Pruning for Large-scale Autonomous Driving Dataset via Trajectory Entropy Maximization

Zhaoyang Liu, Weitao Zhou, Junze Wen, Cheng Jing, Qian Cheng, Kun Jiang, Diange Yang

TL;DR

The paper addresses data-inefficiency in large-scale autonomous driving datasets by introducing trajectory-distribution entropy as an intrinsic measure of scenario diversity and a lightweight, model-agnostic pruning algorithm. By maximizing the entropy of retained trajectory distributions, the method preserves statistical coverage and critical diversity while reducing data volume. Empirical results on NuPlan show up to 40% data reduction without compromising closed-loop performance, highlighting strong practical value for fleet-scale data management and efficient policy learning. The approach offers a scalable, information-theoretic alternative to model-centric or synthetic-data distillation strategies.

Abstract

Collecting large-scale naturalistic driving data is essential for training robust autonomous driving planners. However, real-world datasets often contain a substantial amount of repetitive and low-value samples, which lead to excessive storage costs and bring limited benefits to policy learning. To address this issue, we propose an information-theoretic data pruning method that effectively reduces the training data volume without compromising model performance. Our approach evaluates the trajectory distribution information entropy of driving data and iteratively selects high-value samples that preserve the statistical characteristics of the original dataset in a model-agnostic manner. From a theoretical perspective, we show that maximizing trajectory entropy effectively constrains the Kullback-Leibler divergence between the pruned subset and the original data distribution, thereby maintaining generalization ability. Comprehensive experiments on the NuPlan benchmark with a large-scale imitation learning framework demonstrate that the proposed method can reduce the dataset size by up to 40% while maintaining closed-loop performance. This work provides a lightweight and theoretically grounded approach for scalable data management and efficient policy learning in autonomous driving systems.

Are All Data Necessary? Efficient Data Pruning for Large-scale Autonomous Driving Dataset via Trajectory Entropy Maximization

TL;DR

The paper addresses data-inefficiency in large-scale autonomous driving datasets by introducing trajectory-distribution entropy as an intrinsic measure of scenario diversity and a lightweight, model-agnostic pruning algorithm. By maximizing the entropy of retained trajectory distributions, the method preserves statistical coverage and critical diversity while reducing data volume. Empirical results on NuPlan show up to 40% data reduction without compromising closed-loop performance, highlighting strong practical value for fleet-scale data management and efficient policy learning. The approach offers a scalable, information-theoretic alternative to model-centric or synthetic-data distillation strategies.

Abstract

Collecting large-scale naturalistic driving data is essential for training robust autonomous driving planners. However, real-world datasets often contain a substantial amount of repetitive and low-value samples, which lead to excessive storage costs and bring limited benefits to policy learning. To address this issue, we propose an information-theoretic data pruning method that effectively reduces the training data volume without compromising model performance. Our approach evaluates the trajectory distribution information entropy of driving data and iteratively selects high-value samples that preserve the statistical characteristics of the original dataset in a model-agnostic manner. From a theoretical perspective, we show that maximizing trajectory entropy effectively constrains the Kullback-Leibler divergence between the pruned subset and the original data distribution, thereby maintaining generalization ability. Comprehensive experiments on the NuPlan benchmark with a large-scale imitation learning framework demonstrate that the proposed method can reduce the dataset size by up to 40% while maintaining closed-loop performance. This work provides a lightweight and theoretically grounded approach for scalable data management and efficient policy learning in autonomous driving systems.
Paper Structure (19 sections, 8 equations, 3 figures, 1 table, 1 algorithm)

This paper contains 19 sections, 8 equations, 3 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Heuristic and feature-based Methods
  4. Model-Based Data Valuation
  5. Dataset Distill Methods
  6. Method
  7. Problem Formulation
  8. Data collection.
  9. Model training.
  10. Data pruning objective.
  11. Trajectory Distribution Information Entropy
  12. Data Pruning via Entropy Maximization
  13. Data Pruning via Entropy Maximization
  14. Entropy-Guided Pruning Algorithm
  15. Experiments
  16. ...and 4 more sections

Figures (3)

  • Figure 1: In autonomous driving, vast amounts of naturalistic driving data are collected from real-world operations. However, only a small subset contributes meaningfully to model improvement, while most samples are redundant. Our goal is to automatically identify and retain high-value data in an online and scalable manner, reducing storage and training costs without compromising performance
  • Figure 2: The framework of our method: by calculating the trajectory distribution information entropy, we determine whether the new data on the vehicle side can increase the information entropy value relative to the existing data, thereby achieving fast and efficient data pruning.
  • Figure 3: The visualization results of the trajectory distribution information entropy in the experiments show that the pruned entropy is higher. We can observe that our pruning method effectively removes a large number of repeated stationary and straight trajectories while preserving a smaller proportion of turning trajectories.