What’s a Generative Adversarial Network? Inventor Explains

by Jamie Beckett

What’s a generative adversarial network? If you haven’t yet heard of generative adversarial networks, don’t worry, you will.

The hottest topic in deep learning, GANs, as they’re called, have the potential to create systems that learn more with less help from humans.

Just ask Ian Goodfellow, who hatched the idea for GANs in 2014 when he was still a Ph.D. student at the University of Montreal. Now a research scientist at Google, Goodfellow explained the workings and whys of GANs to a rapt crowd at the GPU Technology Conference last week.

GANs remove one of the biggest obstacles to advancing AI, and particularly deep learning: the huge amount of human effort required.

Generative Adversarial Networks: “Most Interesting Idea in Last 10 Years”

AI pioneer Yann LeCun, who oversees AI research at Facebook, has called GANs “the most interesting idea in the last 10 years in machine learning.”

Typically, a neural network learns to recognize photos of cats, for instance, by analyzing tens of thousands of cat photos. But those photos can’t be used to train networks unless people carefully label what’s pictured in each image. That’s a time-consuming and costly task.

Cop vs. Counterfeiter: GANs Slash Data Needed for Deep Learning

GANs get around this problem by reducing the amount of data needed to train deep learning algorithms. And they provide a unique way to train deep learning algorithms to create labeled data – images, in most cases – from existing data.

Rather than train a single neural network to recognize pictures, researchers train two competing networks. Extending the cat example, a generator network tries to create pictures of fake cats that look like real cats. A discriminator network examines the cat pictures and tries to determine whether they’re real or fake.

“You can think of this being like a competition between counterfeiters and police,” Goodfellow said. “Counterfeiters want to make fake money and have it look real, and police want to look at any particular bill and determine if it’s fake.”

When Fakes Get Real: Competing Neural Networks

The sparring networks learn from each other. As one works hard to find fake images, for example, the other gets better at creating fakes that are indistinguishable from the originals.

NVIDIA founder and CEO Jensen Huang, who described GANs as a “breakthrough” during his GTC keynote, compares the process to an art forger trying to pass off imitations of Picasso paintings as the real thing.

“After training, what you end up with is a network that is able to paint like Picasso, and you have another network that is able to recognize images and paintings at an unheard-of level of discrimination,” he said.

That’s important for fields like medicine, where privacy concerns limit the amount of available data. GANs can fill in the missing data, making it possible to produce entirely fabricated patient datasets that are just as useful for training AI as the real thing.

“You don’t want to put the patient through test after test,” Goodfellow said. “You want to be able to take results of a few tests and generate more.”

How GANs Turn a Horse Into a Zebra

GANs have an artistic side, too.

Want to draw but have no talent? Using a type of generative adversarial network created by researchers at the University of California, Berkeley, you make a rough sketch of what you want, choose colors and instantly turn your scribble into a drawing.

Jun-Yan Zhu, a Ph.D. candidate on that same Berkeley team, demonstrates how to use a GAN to turn a picture of a horse into a zebra, an orange into an apple, a van Gogh painting into a Cezanne, and more.

GANs also generate high-resolution images from low-resolution ones and convert aerial maps into photos, and they make it possible to do all sorts of photo manipulation.

“You can do things like change all kinds of properties in a face – the color of the lips or the arrangement of the hair – but still make sure it remains a realistic face with very sharp color,” Goodfellow said.

Generative Adversarial Network Challenges Remaining

Generative adversarial networks require additional research to reach their potential, Goodfellow said. Sometimes images generated fall short of resembling reality. And GANs are still far from being able to generate complex data.

“We’re really good at making a GAN that can create one kind of image,” he said. “What’s really hard is to create a GAN that can draw dogs and cars and horses and all the images in the world.”