This February, McAfee illuminated the alarming gap between the road test and real-world performance of autonomous vehicle sensors when they had researchers fool a Tesla car into accelerating to 85 mph in a 35 mph zone using a piece of duct tape.
Mobileye, which provides Advanced Driver Assistance Systems (ADAS) for fleets, has deployed these type of cameras in 40 million vehicles, including heavy-duty trucks. I have no doubt both supplier and manufacturer have rigorous test programs in place to ensure safety, but this simple trick clearly illustrates serious implications for the development of future driver-less truck fleets.
There are likely other latent vulnerabilities in vehicle sensor perception systems that we simply have not yet considered, and these could have wider implications as ADAS are given greater responsibility for the operation of vehicles. The scale of the technology challenge that lies ahead in predicting and designing for every eventuality is unprecedented.
New autonomous technologies already being tested for the road freight industry. These include ‘truck platooning’ for up to SAE Automation level 3, where connected convoys are linked by live data exchange. This enables the fleet to share video from the lead vehicle about road hazards ahead, autonomously maintain a set distance from the other trucks and simultaneously brake.
The problem is that any faults in these automated driving support systems could have a dangerous domino effect across a fleet that is electronically synchronized by real-time data. If the lead vehicle camera was duped into exceeding the speed limit, the rest of the convoy could follow, with potentially disastrous effects.
The key question is why latent faults with autonomous vehicle systems are not being picked up during road test campaigns. Autonomous systems are all subjected to extensive and rigorous road tests. Google’s Waymo has conducted 20 million miles of testing on public roads, and Hyundai is rigorously testing its new ‘truck platooning’ on a motorway proving ground designed to replicate real-world conditions.
Because no two tests can replicate the same conditions, many real-world road tests don’t provide realistic validation of autonomous systems. Vehicle cameras may be tested rigorously on many different speed signs, but they are unlikely to be tested on a speed sign covered in duct tape. This demonstrates an important difference between ‘test’ conditions and real-world conditions. Even extensive road tests on public roads cannot easily prepare autonomous vehicle systems for ‘edge cases’ such as malicious actors graffitiing over road signs, precisely because such events are rare.
They also fail to stress-test a vehicle against the full range of normal conditions they are likely to encounter in the real world. For example, a truck rigorously road-tested in daylight hours may not have accounted for the fact that the sun is in a different position at different times of the day, and this affects visibility.
Unlike scientific tests, real-world conditions can never be precisely replicated because no two days are ever the same. Everything from natural to human hazards vary widely from one moment to the next. Even 70 miles driven on the same day will not expose a vehicle to the full range of conditions it may encounter in the same area on a different day.
What should be done
In order to truly test and improve the performance of a vehicle, we need to put it through its paces in extreme as well as ordinary conditions. We need, for example, to test the physical limits of a vehicle on certain road conditions like an icy road, including how slippery conditions might impact braking and steering commands.
When you read that an autonomous vehicle technology has exceeded however many millions of driver-less miles – on the road, or virtually – keep in mind that this is by no means a metric for safety.
What matters is how many ‘smart miles’ it has covered in terms of the range of conditions and scenarios it was exposed to during each mile. Determining which scenarios are required to test the sensor perception, AI and other autonomous systems is now the big challenge we face as an industry to bring safe autonomous freight fleets to fruition.
Pioneering car makers such as Audi are responding to this by taking road-testing off-road and using simulation software to test vehicles in the most extreme driving conditions before they have even been built. With powerful enough simulation software, road tests can be simultaneously conducted across thousands of miles and thousands of variables, dramatically increasing resilience, road-worthiness and safety of new vehicle systems while also reducing the cost of getting them there.
For example, simulations can be used to test ADAS and autonomous driving systems, which enable vehicle dynamics to be assessed in a range of theoretical real-world scenarios to test the physical limits of a vehicle and understand its movement and handling before it rolls onto the roads. Freight vehicles could be tested on ‘virtual’ hills or bumps to analyse how the vibrations from the road affect the on-board sensors so that design engineers can evaluate how the vehicle should respond and whether it can cope with a particular command. Rare edge cases such as electricity blackouts can also be modeled in simulation so that vehicles are roadworthy in all conditions, not just perfect test conditions.
These tests can help uncover unexpected defects with autonomous vehicle systems such as image-recognition cameras that may otherwise not have been detected until it was too late. For example, virtual tests can measure how a truck cab tilts forward when a truck brakes sharply, obstructing the camera’s view and prohibiting the perception system from being able to measure distance accurately and inform the safest ADAS decision. ADAS engineers use this data to improve sensor fusion systems that know which sensor data to trust when.
Simulation’s scope is not limited to vehicles. It also enables autonomous vehicle fleets to be tested in future urban environments to accommodate the widespread introduction of autonomous vehicles for public, private and service transport. In addition to demanding an overhaul of the urban landscape, it will also require sophisticated autonomous systems that understand concepts such as fleet management. This allows both vehicles and city infrastructure scenarios to be safety-tested and improved before either have been constructed.
If we are to ensure the safe and seamless introduction of autonomous vehicles, we clearly need to take the bulk of road testing off-road. Crucially, we need to understand that vehicle safety cannot be measured in distance traveled alone, and no amount of real-world road tests can ever prepare vehicles for the scenarios they could encounter on the road.
Dr. Luca Castignani is MSC Software's automotive industry leader and chief strategist for autonomous driving. An industry veteran with more than 15 years' experience, Castignani designs and implements MSC’s end-to-end solution offerings for ADAS and autonomous driving needs.
