NVIDIA Chief Scientist Bill Dally on How GPUs Ignited AI, and Where His Team’s Headed Next

by Isha Salian

Bill Dally has been working on neural networks since before they were cool.

Dally, chief scientist at NVIDIA, is an icon in the deep learning world. A prolific researcher with more than 150 patents, he previously chaired Stanford University’s computer science department.

Dally sat down with AI podcast host Noah Kravitz to share his reflections on artificial intelligence — a field he’s been working in for decades, which has had a renaissance thanks to GPU-driven deep learning. AI, he says, is “going to transform almost every aspect of human life.”

Roots of the Current AI Revolution

When Dally first started his neural networks research in the 1980s, “we had computers that were literally 100,000 times slower than what we have today,” he told Kravitz.

Today’s AI revolution is enabled by powerful GPUs. But it took a lot of work to get there, such as the 2006 launch of the CUDA programming language by NVIDIA’s Ian Buck.

“The GPUs had the computational resources, and CUDA unlocked it,” Dally said.

As GPU computing gained traction, Dally met with fellow deep learning luminary Andrew Ng for breakfast. Ng was working on a now well-known project that used unsupervised learning to detect images of cats from the web.

This work took 16,000 CPUs on Google Cloud. Dally suggested they collaborate to use GPUs for this work — and so began NVIDIA’s dive into deep learning.

Dally says there are two main focus areas for neural networks going forward: building more powerful algorithms that ramp up the efficiency of doing inference, and developing neural networks that train on much less data.

Technological advancements have an “evolutionary component and a revolutionary component,” he said. “In research, we try to focus on the revolutionary part.”

Strengthening Research Culture at NVIDIA

When Dally joined NVIDIA as chief scientist in 2009, the research team had less than a dozen scientists. Today, it’s 200 strong.

Dally’s goal is for NVIDIA researchers to do excellent work in areas that will have a major impact to the company in the future. He says publishing strong research in top-tier venues is essential because it provides peer review feedback that is key for quality control.

“It’s a humbling experience,” he said. “It makes you better.”

This week, NVIDIA researchers are presenting 14 accepted papers and posters, seven of them during oral sessions, at the annual Computer Vision and Pattern Recognition conference in Salt Lake City.

How to Tune in to the AI Podcast

The AI Podcast is available through iTunes, DoggCatcher, Google Play Music, Overcast, PlayerFM, Podbay, Pocket Casts, PodCruncher, PodKicker, Stitcher and Soundcloud. If your favorite isn’t listed here, email us at aipodcast [at] nvidia [dot] com.