AI Podcast: How Deep Learning Can Translate American Sign Language

Deep learning has accelerated machine translation between spoken and written languages. But it’s lagged behind when it comes to sign language.

Now Syed Ahmed, a computer engineering major at the Rochester Institute of Technology, is unleashing its power to translate between sign language and English.

“You want to talk to your deaf or hard of hearing friend, but you don’t know sign language, what would you do in that case?” says Ahmed, a research assistant at the National Technical Institute for the Deaf, in a conversation with Michael Copeland in this week’s episode of the AI Podcast.

Ahmed fed around 1,700 sign language videos into a deep learning algorithm. The model was able to analyze physical movements and translate it into American English.

“You point your phone at your friend and while they sign, automatic captions appear on your phone,” Ahmed said.

Bridging a gap between a visual language and a written language, the model’s possibilities are endless, according to Ahmed. The AI’s ability to map body language can help predict certain health conditions. In an augmented world, phones wouldn’t be a necessary medium with translated words appearing on the side of the speaker’s face.

Testing is still underway. Since the model is as good as the data set, one of the challenges the system has faced is interpreting colloquial terms in ASL. The amount of videos the algorithm has to sort through also causes the output of captions to lag.

“It’s very new right now, in the future we will be making more experiments with decision making tasks,” Ahmed said.

AI Podcast: Unity’s Danny Lange Hopes to Bring AI to Gaming

Gamer? If you missed last week’s episode, listen to Danny Lange, Unity Technologies’ VP of AI and Machine Learning, discuss how he’s leading efforts to use machine learning to create better games, faster.

How to Tune in to the AI Podcast

The AI Podcast is available through iTunes, DoggCatcher, Google Play Music, Overcast, PlayerFM, Podbay, Pocket Casts, PodCruncher, PodKicker, Stitcher and Soundcloud. If your favorite isn’t listed here, email us at aipodcast [at] nvidia [dot] com.

Listen on Google Play Music itunes

Similar Stories

  • https://clipartpig.com/pencil-clipart-hd-1081 pencil clipart

    Bridging a gap between a visual language and a written language, the model’s possibilities are endless, according to Ahmed. The AI’s ability to map body language can help predict certain health conditions. In an augmented world, phones wouldn’t be a necessary medium with translated words appearing on the side of the speaker’s face.

  • Sarsie

    It would be really great if there were a transcript available so that one of the populations this research purports to involve (the Deaf/hard of hearing community) could access the information and discussion.

  • Henrik Pedersen

    Does anyone know where Syed Ahmed got his data from? He mentions a Youtube database from CCNY (around 4:28), but nothing comes up when I google it. Thanks.