California DMV’s Request For Clarifications Show The Problems With Self-Driving Testing

By Lynn Walford, AUTO CONNECTED CAR NEWS

May 1, 2018

California DMV’s Request for Clarifications Show the Problems with Self-Driving Testing

Consumer Watchdog today backed the Californian Department of Motor Vehicle’s effort that in effect  told eight robot car manufacturers to stop beating around the bush and more clearly tell the public how robot cars being tested on public roads are actually performing.

Waymo (formerly Google’s robot car unit), GM Cruise, Baidu USA, Delphi Automotive, Drive.ai, Nissan, Telenav, and Zoox all currently permitted to test their robot cars on public roads with safety drivers onboard clarified vague information provided in the 2017 autonomous vehicle disengagement reports.

Letters addressed to CA DMV describe these problems

• Baidu’s Disengagement Reports include perception discrepancy, localization error caused drift, object caused braking with traffic behind, systme fault, brake jabs and system fault.
• Delphi’ (Aptiv) s problems included the inability to identify traffic lights, discrepancy between GPS an actual location, other vehicle operated outside of norm and sun blocking out traffic light signals.
• Drive.ai’s problems include poor motions planning, error in motion and discrepancy from data from senor fusions.
• Cruise automation problems included the AV planning to make a turn where there wasn’t enough space, another vehicle pulling intot he way, and the AV plannint to run into oncoming traffic, sensors not capturing the data of vehicles in oncoming vehicles or those entering intersection, or the vehicle not identifying construction zones.
• Nissan reported AV system failures, instances where the AV is about to collide with other vechicle, rear-ending or driver stops AV voluntarily.
• Telenav experienced distance keeping, steering and false positive problems.
• Waymo problems include discrepancies in perceptions, hardware, software and other drivers.
• Zoox problems include limitations of the vehicle planning system, perception discrepancy and hardware descrepancy.

“The companies tried to hide behind technical jargon  and provided limited, vague, and confusing information about robot car performance. It’s great to see the DMV doing its job by requiring the companies that tried to obfuscate important information in their reports to provide supplemental details,” said John M. Simpson, Privacy Project and Technology Director at Consumer Watchdog.

Consumer Watchdog said that the supplemental information in the eight reports further confirmed that self-driving cars cannot actually drive themselves. The details released by the DMV Monday, the nonprofit, nonpartisan group said, corroborate Consumer Watchdog’s analysis of the reports from February 2018.

In the 2017 disengagement reports, companies such as Waymo cited reasons such as “hardware discrepancies,” “unwanted maneuver,” “perception discrepancies,” among others, without providing any clarification about what these terms mean.

“While the disengagement numbers—the number of times a human driver needed to take over the wheel due to failure in the robot car technology—made it apparent that robot car technology is simply not ready for the roads yet, it was difficult to decipher what the reasons for the failure were due to the vague and confusing details companies provided,” said Sahiba K. Sindhu, consumer advocate at Consumer Watchdog.

For example, Waymo’s original reports cited “perception discrepancies” as one of the most frequent reasons for disengagement. But the supplemental reports required by the DMV clarifed the meaning of this term. Wayno said it meant an “event…[where] a component of the vehicle’s perception system (e.g., camera, lidar, radar) fails to detect an object correctly.”

“Requiring that these robot car manufacturers further clarify what the terms they used as reasons for disengagement was a step in the right direction. But the DMV can go even further. The public deserves to know the specifics: how often does a ‘perception discrepancy’ occur due to failures in the cameras as opposed to failures in lidars? The public deserves to know the specifics of how safe these technologies are, and we can’t properly evaluate these technologies without detailed information about their performance,” said Sindhu.

The 2017 disengagement reports released earlier this year revealed that robot cars tested cannot drive more than 5,596 miles in the best-case scenario without a human test driver taking over the wheel. In most cases, the vehicles cannot travel more than a few hundred miles without needing human intervention, Consumer Watchdog noted.