Software-Hardware Co-optimization for Modular E2E AV Paradigm: A Unified Framework of Optimization Approaches, Simulation Environment and Evaluation Metrics

TL;DR

This work targets the deployment challenges of modular end-to-end (ME2E) autonomous driving by proposing a software–hardware co-optimization framework that jointly compresses models and accelerates computation under a unified system objective. It introduces a real-time synchronous (RTS) CARLA-based simulation platform and a multidimensional evaluation metric, $EER_{AV}$, to jointly quantify safety, comfort, efficiency, latency, and energy, addressing gaps in traditional open-loop and single-dimension assessments. Through module-wise pruning and quantization, plus hardware graph and kernel optimizations in TensorRT, the framework achieves substantial latency and energy reductions while preserving driving performance, with up to $6\times$ latency reduction, roughly $1/5$ per-frame energy, and up to $22.35\%$ improvement in $EER_{AV}$. Key insights include the non-monotonic relationship between inference speed and driving quality, the critical role of latency stability over average latency, and the necessity of coordinated software–hardware design for ME2E deployability. The results offer practical guidance for deploying ME2E systems on edge platforms and suggest avenues for extending the approach to broader intelligent systems and multi-agent scenarios.

Abstract

Modular end-to-end (ME2E) autonomous driving paradigms combine modular interpretability with global optimization capability and have demonstrated strong performance. However, existing studies mainly focus on accuracy improvement, while critical system-level factors such as inference latency and energy consumption are often overlooked, resulting in increasingly complex model designs that hinder practical deployment. Prior efforts on model compression and acceleration typically optimize either the software or hardware side in isolation. Software-only optimization cannot fundamentally remove intermediate tensor access and operator scheduling overheads, whereas hardware-only optimization is constrained by model structure and precision. As a result, the real-world benefits of such optimizations are often limited. To address these challenges, this paper proposes a reusable software and hardware co-optimization and closed-loop evaluation framework for ME2E autonomous driving inference. The framework jointly integrates software-level model optimization with hardware-level computation optimization under a unified system-level objective. In addition, a multidimensional evaluation metric is introduced to assess system performance by jointly considering safety, comfort, efficiency, latency, and energy, enabling quantitative comparison of different optimization strategies. Experiments across multiple ME2E autonomous driving stacks show that the proposed framework preserves baseline-level driving performance while significantly reducing inference latency and energy consumption, achieving substantial overall system-level improvements. These results demonstrate that the proposed framework provides practical and actionable guidance for efficient deployment of ME2E autonomous driving systems.

Figures (6)

• Figure 1: Overview of co-optimization and evaluation framework
• Figure 2: Multi-stage hardware optimization framework for computation graphs and kernels
• Figure 3: Implementation workflow of the RTS real-time synchronous simulation framework
• Figure 4: The impact of real-time performance on closed-loop driving metrics
• Figure 5: Visualization of vehicle trajectories at the accident moment, with the top showing a correct prediction and the bottom showing an incorrect prediction