Learned Path Planning and Vehicle Control

Dr. Bernhard Finer


In our NVIDIA lab in New Jersey we taught a deep convolutional neural network (DNN) to drive a car by observing human drivers and emulating their behavior. We found that these networks can learn more aspects of the driving task than is commonly learned today. We present examples of learned lane keeping, lane changes, and turns. We also introduce tools to visualize the internal information processing of the neural network and discuss the results. Work on our deep convolutional nets started in the spring of 2015 with a core team based in Holmdel, New Jersey. By March 2016 the team could demonstrate end-to-end learning of the complete control loop from cameras to steering actuation. A demo system that drove on local roads and highways was successfully trained with about 70 hours of driving data. In the past year the team has grown and capabilities of our system have expanded, now including learned lane changes and turns at intersections. In addition, a simulator was built that faithfully reproduces real vehicle behavior.