Computer-generated special effects have amazed audiences for nearly a generation now. But they’ve long fallen short when modeling the one thing that’s most important to telling a good story: the human face. Until now.
At our annual GPU Technology Conference this week, Jen-Hsun Huang, our CEO, showed off a startling digital model of a human head that not only offered unparalleled realism — but was generated in real-time using commercially-available GPUs and an NVIDIA innovation we call FaceWorks.
“Let’s have Ira say a few things,” Jen-Hsun asks during his keynote presentation, as Ira’s visage loomed overhead. “What did you have for breakfast?”
Ira complains that he had yogurt for breakfast, but it was largely frozen fruit.
Asked about Project SHIELD, Ira screams “take my money!”
To create ‘Ira’ we partnered with Dr. Paul Debevec of the Institute for Creative Technology (ICT) at the University of Southern California. He and his team have been building next-generation systems that can capture facial data to within a tenth of a millimeter without the need for special makeup, awkward markers or specialized cameras.
ICT’s ‘light stage’ technology uses photographic techniques to derive not only the 3D shape of an actor’s face, but the critical elements needed to properly represent human skin. These include light reflection and transmission through the skin, reflectivity from oils in the skin, and the nearly microscopic lines and bumps in the skin surface.
After asking an actor to model several dozen basic expressions, Debevec’s team then runs their footage through a compression engine. The result: a pared down set of data that can be mixed and matched to generate a wide range of expressions.
Enter the GPU. All the incredible detail gathered by Debevec’s team is for naught if it can’t be processed and displayed on the screen interactively. NVIDIA’s FaceWorks technology running on the world’s fastest GPUs accomplish exactly this.
All the subtlety of ‘crows feet’ around the eyes, light transmission through ears and nose, and the micro bumps on the skin surface are precisely represented and interactively manipulated on the screen.
That’s not simple. Complex skin shaders executing over 8,000 instructions per pixel allow realistic rendering of the face in any lighting situation. At full HD resolution that’s 82 billion floating point operations per second (FLOPs) per frame – or 4.9 trillion FLOPs per second at 60 frames per second. And all that doesn’t include the 161 filtered texture fetches per pixel.
All this detail comes at a price: size. The data from Debevec’s team exceeds 32 gigabytes for a few expressions. Even the most advanced GPUs will choke on the immensity of that. FaceWorks reduces this to a very manageable 300 megabytes.
The trick is to take advantage of similarities in the textures for each of the actor’s expressions. Each texture is divided into an eight by eight grid of tiles, and those tiles that are recognized as being nearly identical to the same tile in other expressions are discarded.
‘Ira’ makes his appearance at NVIDA’s annual GPU Technology Conference.
The result: a pared down set of meshes and textures that can be mixed and matched to generate a wide range of expressions. Clever, no?
I guess you could say we’re, um, using our heads.
