Waymo Develops New Benchmark for Comparing Robotaxis to Human Drivers

Waymo has developed a new computer model, called the Reference Driver, to more accurately compare its autonomous driving software to human driving capabilities. The model, created in conjunction with TU Delft, uses a framework called active inference, which assumes a driver constantly imagines possible futures and takes actions to reach the safest, most predictable one. The Alphabet-owned robotaxi company published a research paper about the model in Nature Communications on Wednesday.

Waymo expects the new model to be more accurate than its previous version, which it has used over the past several years. The new model will help Waymo better understand how humans behave in crash scenarios that its robotaxis encounter. For decades, the automotive industry has used physical and virtual crash dummies to evaluate a car's safety features.

The new model evolves this concept, serving as a behavioral benchmark for autonomous driving systems able to realistically represent reasonable expectations on how a careful and competent human driver responds to traffic conflicts. A more accurate model of human driving behavior is crucial for autonomous vehicle companies that need to understand and grade the performance of its robotaxis in crashes. This comes at a critical juncture for Waymo, which is scaling to more cities and facing greater scrutiny from regulators and the public.

In January, a Waymo robotaxi struck a child near a school in Santa Monica, California, and the company relied on its previous computer model to claim that an attentive human driver would have made impact at around 14 miles per hour. The biggest difference between the Reference Driver and its predecessor is that it can reproduce a human driver's behavior in the run-up to a crash. Previously, Waymo's models focused on replicating 'last-second, reactive' human maneuvers.

The Reference Driver can simulate the internal 'surprise' a driver feels during a conflict, providing a more human-like benchmark for autonomous driving systems. Waymo says this new driver model can be adapted to model a 'wide range of road user behaviors beyond collision avoidance,' and that it is better-equipped to be applied to 'large test sets with thousands of scenarios.' The model can represent and evaluate numerous complex, real-world crashes in a virtual environment, identifying performance improvements with unprecedented speed and efficiency. The company is making the research code for the model available under an academic, non-commercial license that allows it to be used for research, teaching, personal experimentation, and scientific publication.

Why this matters: The development of the Reference Driver has significant implications for the autonomous vehicle industry. With more accurate assessments of robotaxi performance, companies like Waymo can better identify areas for improvement and optimize their systems. This, in turn, can lead to increased public trust and adoption of autonomous vehicles.