It’s not every day that one of the world’s leading tech companies highlights the benefits of your products.
Intel did just that last week, comparing the inference performance of two of their most expensive CPUs to NVIDIA GPUs.
To achieve the performance of a single mainstream NVIDIA V100 GPU, Intel combined two power-hungry, highest-end CPUs with an estimated price of $50,000-$100,000, according to Anandtech. Intel’s performance comparison also highlighted the clear advantage of NVIDIA T4 GPUs, which are built for inference. When compared to a single highest-end CPU, they’re not only faster but also 7x more energy-efficient and an order of magnitude more cost-efficient.
Inference performance is crucial, as AI-powered services are growing exponentially. And Intel’s latest Cascade Lake CPUs include new instructions that improve inference, making them the best CPUs for inference. However, it’s hardly competitive with NVIDIA deep learning-optimized Tensor Core GPUs.
Inference (also known as prediction), in simple terms, is the “pattern recognition” that a neural network does after being trained. It’s where AI models provide intelligent capabilities in applications, like detecting fraud in financial transactions, conversing in natural language to search the internet, and predictive analytics to fix manufacturing breakdowns before they even happen.
While most AI inference today happens on CPUs, NVIDIA Tensor Core GPUs are rapidly being adopted across the full range of AI models. Tensor Core, a breakthrough innovation has transformed NVIDIA GPUs to highly efficient and versatile AI processors. Tensor Cores do multi-precision calculations at high rates to provide optimal precision for diverse AI models and have automatic support in popular AI frameworks.
It’s why a growing list of consumer internet companies — Microsoft, Paypal, Pinterest, Snap and Twitter among them — are adopting GPUs for inference.
Compelling Value of Tensor Core GPUs for Computer Vision
First introduced with the NVIDIA Volta architecture, Tensor Core GPUs are now in their second generation with NVIDIA Turing. Tensor Cores perform extremely efficient computations for AI for a full range of precision — from 16-bit floating point with 32-bit accumulate to 8-bit and even 4-bit integer operations with 32-bit accumulate.
They’re designed to accelerate both AI training and inference, and are easily enabled using automatic mixed precision features in the TensorFlow and PyTorch frameworks. Developers can achieve 3x training speedups by adding just two lines of code to their TensorFlow projects.
On computer vision, as the table below shows, when comparing the same number of processors, the NVIDIA T4 is faster, 7x more power-efficient and far more affordable. NVIDIA V100, designed for AI training, is 2x faster and 2x more energy efficient than CPUs on inference.
Table 1: Inference on ResNet-50.
| Two-Socket Intel Xeon 9282 | NVIDIA V100 (Volta) | NVIDIA T4 (Turing) |
|
|---|---|---|---|
| ResNet-50 Inference (images/sec) | 7,878 | 7,844 | 4,944 |
| # of Processors | 2 | 1 | 1 |
| Total Processor TDP | 800 W | 350 W | 70 W |
| Energy Efficiency (Taking TDP) | 10 img/ sec/W | 22 img/ sec/W | 71 img/ sec/W |
| Performance per Processor (images/sec) | 3,939 | 7,844 | 4,944 |
| GPU Performance Advantage | 1.0 (baseline) | 2.0x | 1.3x |
| GPU Energy-Efficiency Advantage | 1.0 (baseline) | 2.3x | 7.2x |
Source: Intel Xeon performance; NVIDIA GPU performance
Compelling Value of Tensor Core GPUs for Understanding Natural Language
AI has been moving at a frenetic pace. This rapid progress is fueled by teams of AI researchers and data scientists who continue to innovate and create highly accurate and exponentially more complex AI models.
Over four years ago, computer vision was among the first applications where AI from Microsoft was able to perform at superhuman accuracy using models like ResNet-50. Today’s advanced models perform even more complex tasks like understanding language and speech at superhuman accuracy. BERT, a highly complex AI model open-sourced by Google last year, can now understand prose and answer questions with superhuman accuracy.
A measure of the complexity of AI models is the number of parameters they have. Parameters in an AI model are the variables that store information the model has learned. While ResNet-50 has 25 million parameters, BERT has 340 million, a 13x increase.
On an advanced model like BERT, a single NVIDIA T4 GPU is 59x faster than a dual-socket CPU server and 240x more power-efficient.
Table 2: Inference on BERT. Workload: Fine-Tune Inference on BERT Large dataset.
| Dual Intel Xeon Gold 6240 | NVIDIA T4 (Turing) |
|
|---|---|---|
| BERT Inference, Question-Answering (sentences/sec) | 2 | 118 |
| Processor TDP | 300 W (150 Wx2) | 70 W |
| Energy Efficiency (using TDP) | 0.007 sentences/ sec/W | 1.7 sentences/ sec/W |
| GPU Performance Advantage | 1.0 (baseline) | 59x |
| GPU Energy-Efficiency Advantage | 1.0 (baseline) | 240x |
CPU server: Dual-socket Xeon Gold 6240@2.6GHz; 384GB system RAM; FP32 precision; with Intel’s TF Docker container v. 1.13.1. Note: Batch-size 4 results yielded the best CPU score.
GPU results: T4: Dual-socket Xeon Gold 6240@2.6GHz; 384GB system RAM; mixed precision; CUDA 10.1.105; NCCL 2.4.3, cuDNN 7.5.0.56, cuBLAS 10.1.105; NVIDIA driver 418.67; on TensorFlow using automatic mixed precision and XLA compiler; batch-size 4 and sequence length 128 used for all platforms tested.
Compelling Value of Tensor Core GPUs for Recommender Systems
Another key usage of AI is in recommendation systems, which are used to provide relevant content recommendations on video sharing sites, news feeds on social sites and product recommendations on e-commerce sites.
Neural collaborative filtering, or NCF, is a recommender system that uses the prior interactions of users with items to provide recommendations. When running inference on the NCF model that is a part of the MLPerf 0.5 training benchmark, NVIDIA T4 brings 10x more performance and 20x higher energy efficiency than CPUs.
Table 3: Inference on NCF.
| Single Intel Xeon Gold 6240 | NVIDIA T4 (Turing) |
|
|---|---|---|
| Recommender Inference Throughput (MovieLens)(thousands of samples/sec) | 2,860 | 27,800 |
| Processor TDP | 150 W | 70 W |
| Energy Efficiency (using TDP) | 19 samples/ sec/W | 397 samples/ sec/W |
| GPU Performance Advantage | 1.0 (baseline) | 10x |
| GPU Energy-Efficiency Advantage | 1.0 (baseline) | 20x |
CPU server: Single-socket Xeon Gold 6240@2.6GHz; 384GB system RAM; Used Intel Benchmark for NCF on TensorFlow with Intel’s TF Docker container version 1.13.1; FP32 precision. Note: Single-socket CPU config used for CPU tests as it yielded a better score than dual-socket.
GPU results: T4: Single-socket Xeon Gold 6140@2.3GHz; 384GB system RAM; CUDA 10.1.105; NCCL 2.4.3, cuDNN 7.5.0.56, cuBLAS 10.1.105; NVIDIA driver 418.40.04; on TensorFlow using automatic mixed precision and XLA compiler; batch-size: 2,048 for CPU, 1,048,576 for T4; precision: FP32 for CPU, mixed precision for T4.
Unified Platform for AI Training and Inference
The use of AI models in applications is an iterative process designed to continuously improve their performance. Data scientist teams constantly update their models with new data and algorithms to improve accuracy. These models are then updated in applications by developers.
Updates can happen monthly, weekly and even on a daily basis. Having a single platform for both AI training and inference can dramatically simplify and accelerate this process of deploying and updating AI in applications.
NVIDIA’s data center GPU computing platform leads the industry in performance by a large margin for AI training, as demonstrated by the standard AI benchmark, MLPerf. And the NVIDIA platform provides compelling value for inference, as the data presented here attests. That value increases with the growing complexity and progress of modern AI.
To help fuel the rapid progress in AI, NVIDIA has deep engagements with the ecosystem and constantly optimizes software, including key frameworks like TensorFlow, Pytorch and MxNet as well as inference software like TensorRT and TensorRT Inference Server.
NVIDIA also regularly publishes pre-trained AI models for inference and model scripts for training models using your own data. All of this software is freely made available as containers, ready to download and run from NGC, NVIDIA’s hub for GPU-accelerated software.
Get the full story about our comprehensive AI platform.