Beyond Star Trek’s VISOR: How a GPU-Powered Visual Aid Allows the Blind to ‘See’
Editor’s note: This is one of five profiles of finalists for NVIDIA’s 2017 Global Impact Award, which provides $150,000 to researchers using NVIDIA technology for groundbreaking work that addresses social, humanitarian and environmental problems.
Another futuristic gadget from the world of Star Trek fantasy is a big step closer to becoming reality thanks to pioneering computer vision and prosthetics work by researchers at the University of Oxford.
In the sci-fi show, a blind character wears a VISOR, a device that wraps over his eyes and lets him “see.” Oxford professor Philip Torr, colleague Stephen Hicks and a team of researchers are helping the partially sighted achieve something similar — and potentially transforming how they interact with the world around them.
With 285 million people around the globe affected by impaired vision, per World Health Organization figures, the stakes are high.
The Oxford team’s “visor” is powered by advanced computer vision techniques that use GPUs to process camera images of a user’s environment. These images are then transformed into a visualization, and made available through the lenses of a pair of glasses the team has developed.
The effort has placed the team among five finalists for NVIDIA’s 2017 Global Impact Award. Our annual grant of $150,000 is given to researchers using NVIDIA technology for groundbreaking work that addresses social, humanitarian and environmental problems.
Creating a Visual Map
Scanning a new environment, such as a room, is easy for a person of normal vision. One’s eyes intuitively record the boundaries, note major objects and form a visual map before picking out smaller details.
Such a map is much harder for the visually impaired to construct. There’s little visual information to go on, which can limit their independence and freedom.
“Computer vision offers many opportunities to improve on how someone participates in their everyday environment,” said Torr, who, as a child, was fascinated by robots and artificial intelligence. It also poses many challenges because of high compute needs.
A key aspect to a successful visual prosthetic is allowing users to quickly identify and locate relevant objects that they encounter — to create a visual map. This requires vast amounts of computation to optimize the large number of parameters found in a given setting. To perform this training, the Oxford team focused on efficiently processing data in parallel.
“GPUs are the mainstay of what we’ve been using,” Torr said, who led the computer vision work using an NVIDIA TITAN X. “The GPU architecture and parallel processing are essential for computer vision work. We use the CUDA framework, while developer support allows a lot of the coding we do.”
Integrating computer vision abilities into visual prosthetics is also daunting.
For a user to wear a headset or glasses in an everyday environment, the visual prosthetic must be mobile and lightweight while operating complex algorithms in real time. NVIDIA Tegra GPUs contribute most of the computer power in the team’s prototypes.
Oxford’s Hicks, whose team focuses on visual prosthetics, developed so-called smart glasses for visual impairment. Results from a recently completed U.K.-wide trial of 300 registered blind individual using the glasses were life-changing for some. Up to 30 percent of participants found an immediate significant boost in their ability to see objects. See Hick’s Ted talk, here.
The trial was so successful, Oxsight Ltd., a spinoff company, was formed last year to develop the glasses commercially. The team is now looking at how to use AI in its work.
This could include ways to personalize glasses, so wearing them builds information from the user’s environment, helping the glasses learn to recognize commonplace things.
Other Global Impact Award 2017 finalists include:
Check out the work of last year’s Global Impact Award winner.