The final push for the hat trick came down to the wire.
Five minutes before the deadline, the team submitted work in its third and hardest data science competition of the year in recommendation systems. Called RecSys, it’s a relatively new branch of computer science that’s spawned one of the most widely used applications in machine learning, one that helps millions find what they want to watch, buy and play.
The team’s combination of six AI models packed into the contest’s limit of 20 gigabytes all of the smarts it culled from studying 750 million data points. An unusual rule in the competition said the models had to run in less than 24 hours on a single core in a cloud CPU.
They hit the submission button and waited.
Twenty-three hours and 40 minutes later an email arrived: They hit No. 1 on the leaderboard.
Right Under the Buzzer
On June 28 it was official, a seven-member NVIDIA team won for the second time the ACM RecSys Challenge.
“The email came in right under the buzzer — 20 minutes later and we would have timed out,” said Chris Deotte, one of several team members who’s also a grandmaster in Kaggle competitions, the online Olympics of data science.
“We were really on the edge,” said Benedikt Schifferer, a teammate who helps design NVIDIA Merlin, a framework to help users quickly build their own recommendation systems.
GPUs could have busted through the inference job in a fraction of the time. Adapting the work to one CPU core “was like going back to the distant past,” said Gilberto “Giba” Titericz, a Brazil-based Kaggle grandmaster on the team.
In fact, once the competition was over, the team demonstrated the inference job that took nearly 24 hours on a CPU core could run on a single NVIDIA A100 Tensor Core GPU in just five and a half minutes.
Sorting 40M Items a Day
For that competition, Twitter gave participants millions of data points a day for 28 days and asked them to predict which tweets users would like or retweet. It was an industrial-strength challenge from the leading technical conference on RecSys, an event that draws a who’s who of top engineers from Facebook, Google, Spotify and other players.
The discipline is as hard as it is helpful. Recommendation systems fuel our digital economy, serving up suggestions faster and smarter than a traditional search.
Industry challenges help advance the field for everyone, whether they’re seeking the perfect gift for a spouse or trying to find an old friend online.
Three Wins in Five Months
Earlier this year, the full NVIDIA team led a field of 40 in the Booking.com Challenge. They used millions of anonymized data points to correctly predict the final city a vacationer in Europe would choose to visit.
In June, another top recsys contest, the SIGIR eCommerce Data Challenge, set an even higher hurdle.
The annual meeting of the Special Interest Group on Information Retrieval, SIGIR, draws experts from companies that span Alibaba to Walmart Labs. Its 2021 challenge provided 37 million data points from online shopping sessions and asked participants to predict which products users would buy.
Overlap with the ACM contest forced the NVIDIA team to split into two groups that coordinated their efforts between the contests. Ratcheting up the pressure, some team members were heads down writing a paper for the ACM RecSys conference.
The Art of the Fast Break
Two factors propelled a five-person NVIDIA team with members spread across Brazil, Canada, France and the U.S. to the best overall performance, taking first or second place in every leaderboard. They made a big bet on Transformer models developed for natural-language processing and increasingly adopted for recsys, and they understood the art of the handoff.
“As one member is going to bed another picks up the work in a different time zone,” said Even Oldridge, who leads the Merlin group.
“When it all clicks, it’s very effective, and I’m amazed at what we’ve accomplished in the last year building our internal knowledge and our standing in the recsys community to the point where we could win three major competitions in five months,” he said.
Respecting User Privacy
The contest required models to make predictions with no background on users beyond their current browsing session.
“That’s an important task because sometimes users want to browse anonymously, and some privacy laws limit access to historical information,” said Gabriel Moreira, a senior Merlin researcher in São Paulo who led NVIDIA’s SIGIR team.
The competition marked the first time the team used only Transformer models in their solution to a challenge. Moreira’s team aims to make the massive neural networks more easily available to every Merlin customer.
From a Hat Trick to a Haul
On June 30, we notched a fourth consecutive win in RecSys, what hockey players call a haul. MLPerf, an industry benchmarking group, announced that NVIDIA and its partners set records in all its latest training benchmarks, including one in recommendation systems.
The team behind that effort described its work training a recommendation system in less than a minute on 14 NVIDIA DGX systems, a 3.3x speedup compared to their submission a year ago.
Sharing Lessons Learned
The competitions fuel ideas for new techniques that find their way into recsys frameworks like Merlin and related tools, papers and online classes held by the NVIDIA Deep Learning Institute. The ultimate goal: Help everyone succeed.
In interviews NVIDIA’s recsys experts freely shared their know-how — part art, part science.
A Pro Tip on RecSys
One best practice is using a diversity of models that work together as an ensemble.
In the ACM RecSys Challenge, the team used both tree and neural-network models. The outputs from one stage became inputs for the next in a process called stacking.
“A single model can make a mistake due to a data error or convergence issue, but if you take an ensemble of several models, it’s very powerful,” said Bo Liu, the newest member of NVIDIA’s Kaggle grandmaster team.
Meet RecSys Experts Online
On July 29, you can meet RecSys experts from Facebook, NVIDIA and TensorFlow to learn more about how to create great recommender systems.