Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Comparison of Waymo Rider-Only Crash Data to Human Benchmarks at 7.1 Million Miles

Kristofer D. Kusano, John M. Scanlon, Yin-Hsiu Chen, Timothy L. McMurry, Ruoshu Chen, Tilia Gode, Trent Victor

Abstract

This paper examines the safety performance of the Waymo Driver, an SAE level 4 automated driving system (ADS) used in a rider-only (RO) ride-hailing application without a human driver, either in the vehicle or remotely. ADS crash data was derived from NHTSA's Standing General Order (SGO) reporting over 7.14 million RO miles through the end of October 2023 in Phoenix, AZ, San Francisco, CA, and Los Angeles, CA. When considering all locations together, the any-injury-reported crashed vehicle rate was 0.6 incidents per million miles (IPMM) for the ADS vs 2.80 IPMM for the human benchmark, an 80% reduction or a human crash rate that is 5 times higher than the ADS rate. Police-reported crashed vehicle rates for all locations together were 2.1 IPMM for the ADS vs. 4.68 IPMM for the human benchmark, a 55% reduction or a human crash rate that was 2.2 times higher than the ADS rate. Police-reported and any-injury-reported crashed vehicle rate reductions for the ADS were statistically significant when compared in San Francisco and Phoenix, as well as combined across all locations (except for any-injury-reported in Phoenix). The any property damage or injury comparison had statistically significant decrease in 3 comparisons, but also non-significant results in 3 other benchmarks. Given imprecision in the benchmark estimate and multiple potential sources of underreporting biasing the benchmarks, caution should be taken when interpreting the results of the any property damage or injury comparison. Together, these crash-rate results should be interpreted as a directional and continuous confidence growth indicator, together with other methodologies, in a safety case approach.

Comparison of Waymo Rider-Only Crash Data to Human Benchmarks at 7.1 Million Miles

Abstract

This paper examines the safety performance of the Waymo Driver, an SAE level 4 automated driving system (ADS) used in a rider-only (RO) ride-hailing application without a human driver, either in the vehicle or remotely. ADS crash data was derived from NHTSA's Standing General Order (SGO) reporting over 7.14 million RO miles through the end of October 2023 in Phoenix, AZ, San Francisco, CA, and Los Angeles, CA. When considering all locations together, the any-injury-reported crashed vehicle rate was 0.6 incidents per million miles (IPMM) for the ADS vs 2.80 IPMM for the human benchmark, an 80% reduction or a human crash rate that is 5 times higher than the ADS rate. Police-reported crashed vehicle rates for all locations together were 2.1 IPMM for the ADS vs. 4.68 IPMM for the human benchmark, a 55% reduction or a human crash rate that was 2.2 times higher than the ADS rate. Police-reported and any-injury-reported crashed vehicle rate reductions for the ADS were statistically significant when compared in San Francisco and Phoenix, as well as combined across all locations (except for any-injury-reported in Phoenix). The any property damage or injury comparison had statistically significant decrease in 3 comparisons, but also non-significant results in 3 other benchmarks. Given imprecision in the benchmark estimate and multiple potential sources of underreporting biasing the benchmarks, caution should be taken when interpreting the results of the any property damage or injury comparison. Together, these crash-rate results should be interpreted as a directional and continuous confidence growth indicator, together with other methodologies, in a safety case approach.
Paper Structure (5 sections, 3 equations, 3 figures, 7 tables)

This paper contains 5 sections, 3 equations, 3 figures, 7 tables.

Table of Contents

  1. Appendix
  2. Selection of SGO Events
  3. Results for Los Angeles
  4. Example of nelson1970confidence Rate Ratio Confidence Intervals
  5. Considerations for Safety Impact Assessment for Serious Injury and Fatal Crashes

Figures (3)

  • Figure 1: Visualization of Waymo’s Safety Determination Life Cycle (adapted from Figure 6, favaro2023building)
  • Figure 2: Conceptual Relationship Between Different Crash Severity Reporting Thresholds. The arrows below the graph show the hypothetical range of severities included in each outcome group.
  • Figure 3: Percent Reduction for the Waymo ADS in Police-Reported and Any-Injury-Reported Crashes Compared to a Human Benchmark from scanlon2023benchmark in Phoenix, San Francisco, and All Locations. Note that Los Angeles data is not plotted because of large error bars. See the results in the Appendix