Want Drones to Fly Right? Make Them Hallucinate, MIT Prof Says

by Noah Kravitz

How do you teach a drone to fly better? Make it hallucinate.

Okay. So how do you make a drone hallucinate? With VR, of course.

Still with us? Good. Let’s explain.

MIT’s Sertac Karaman has been building drones and teaching them to fly autonomously for awhile now. As associate professor of aeronautics and astronautics, and director of the Foundations of Autonomous Systems Technology Group at MIT, he focuses on research on autonomous vehicles and transportation of all sorts, on the ground and in the air.

Karaman and his colleagues just built their second generation agile drone based on the NVIDIA Jetson embedded computing platform. And they found a novel way to train it.

Inspired by police in the Netherlands who trained falcons to take down drones flying unsafely, Karaman and his colleagues sought to study how the birds of prey do it. They affixed cameras to the backs of the birds and filmed them hunting.

Sertac Karaman's first and second-generation agile drones.
Sertac Karaman’s first and second-generation agile drones.

The video revealed how falcons respond to visual cues while diving at speeds of 70 meters per second, turning their heads mid-flight to better track their prey.

Based on this research, Karaman’s team built “Penguin.” It’s a fully autonomous indoor drone with high-speed cameras that feed image streams to a Jetson supercomputer.

But that was back in the days of Penguin. Today it’s all about Sparrow, MIT’s second-gen agile drone based on Jetson TX2. With three onboard cameras and an xSens Precision IMU for 3D motion tracking, Sparrow runs real-time visual SLAM to navigate as it flies.

MIT’s second-gen agile drone is based on Jetson TX2.

Which brings us to hallucinations.

Using an NVIDIA TITAN X-based workstation for photorealistic rendering, Karaman can fly Sparrow in an empty motion capture room while feeding it simulated environments to navigate. Flying through the simulations — what Karaman calls “hallucinations” — the drone learns to navigate around obstacles like those it might encounter on a rescue mission or working in a factory.

When Sparrow crashes, it’s penalized within the training environment, and then stabilized and set back to flying. When the drone successfully completes a mission, it’s rewarded. “In the future, we’re hoping to use our DGX-1 to take all the data and run reinforcement learning,” Karaman said.

Similar to using VR to safely train humans for dangerous work, making Sparrow hallucinate lets Karaman’s team train the drone without risking hardware damage in real-world collisions.

All work and no play makes Sparrow a dull bird. So Karaman and team are also teaching Sparrow to race autonomously and chase other drones.

The team additionally has its sights set on developing a next-generation drone that relies on a Jetson TX2 for all computing functions, including real-time computing and CPU functions. “The more performance the better, of course,” Karaman said. “Jetson TX2 delivers.”

For a closer look at the technology, check out Karaman’s video below: