Driver’s Ed for Self-Driving Cars: How Our Deep Learning Tech Taught a Car to Drive

by Danny Shapiro

For many of us, learning to drive is a rite of passage to adulthood. At first, you’re jittery and overly cautious, but as the miles pass you get better. You learn to understand the nuances, how the elements affect your trajectory and how to adjust.

A team of engineers from NVIDIA based in our New Jersey office — a former Bell Labs office that also happens to be the birthplace of the deep learning revolution currently sweeping the technology industry — decided that they would use deep learning to teach an autonomous car to drive. They used a convolutional neural network (CNN) to learn the entire processing pipeline needed to steer an automobile.

The project, called DAVE2, is part of an effort kicked off nine months ago at NVIDIA to build on the DARPA Autonomous Vehicle (DAVE) research to create a robust system for driving on public roads. We wanted to bypass the need to hardcode detection of specific features — such as lane markings, guardrails or other cars — and avoid creating a near infinite number of “if, then, else” statements, which is too impractical to code when trying to account for the randomness that occurs on the road.

So how did our test car learn to drive?

Using the NVIDIA DevBox and Torch 7 (a machine learning library) for training, and an NVIDIA DRIVE PX self-driving car computer to process it all, our team trained a CNN with time-stamped video from a front-facing camera in the car synced with the steering wheel angle applied by the human driver.

We collected the majority of the road data  in New Jersey, including two-lane roads with and without lane markings, residential streets with parked cars, tunnels and even unpaved pathways. More data was collected in clear, cloudy, foggy, snowy and rainy weather, both day and night.

NVIDIA self-driving car, taught with deep learning
NVIDIA used deep learning to create a self-driving car.

Using this data, our team trained a CNN to steer the same way a human did given a particular view of the road and evaluated in simulation. Our simulator took videos from the data-collection vehicle and generated images that approximate what would appear if the CNN were instead steering the vehicle.

Once the trained CNN showed solid performance in the simulator, we  loaded it onto DRIVE PX and took it out for a road test in the car. The vehicle drove along paved and unpaved roads with and without lane markings, and handled a wide range of weather conditions. As more training data was gathered, performance continually improved. The car even flawlessly cruised the Garden State Parkway.

Our engineering team never explicitly trained the CNN to detect road outlines. Instead, using the human steering wheel angles versus the road as a guide, it began to understand the rules of engagement between vehicle and road.

For more details, check out the NVIDIA research paper “End to End Learning for Self-Driving Cars.”