NVIDIA has released updated capabilities for robotics AI perception and simulation, with a new version of the NVIDIA Isaac software development kit.
Announced by company founder and CEO Jensen Huang at NVIDIA’s latest GPU Technology Conference, the SDK achieves an important milestone in establishing a unified robotic development platform — enabling AI, simulation and manipulation capabilities.
It includes the Isaac Robotics Engine (which provides the application framework), Isaac GEMs (pre-built deep neural networks models, algorithms, libraries, drivers and APIs), reference apps for indoor logistics, as well as the first release of Isaac Sim (offering navigation capabilities).
As a result, the new Isaac SDK can greatly accelerate developing and testing robots for researchers, developers, startups and manufacturers. It enables the AI-powered perception and training of robots in simulation — allowing them to be tested and validated in a range of environments and situations.
And in doing so, it saves costs.
Start with AI-Based Perception
Every autonomous machine starts with perception.
To jumpstart AI robotic development, the new Isaac SDK includes a variety of camera-based perception deep neural networks. Among them:
- Object detection — recognizes objects for navigation, interaction or manipulation
- Free space segmentation — detects and segments the external world, such as determining where a sidewalk is and where a robot is allowed to travel
- 3D pose estimation — understands an object’s position and orientation, enabling such tasks as a robotic arm picking up an object
- 2D human pose estimation — applies pose estimation to humans, which is important for robots that interact with people, such as delivery bots, and for cobots, which are specifically designed to work together with humans
The SDK’s object detection has also been updated with the ResNet deep neural network, which can be trained using NVIDIA’s TAO Toolkit. This makes it easier to add new objects for detection and train new models that can get up and running with high accuracy levels.
Introducing Isaac Sim
The new release introduces an important capability — using Isaac Sim to train a robot and deploy the resulting software into a real robot that operates in the real world. This promises to greatly accelerate robotic development, enabling training with synthetic data.
Simulation enables testing in so-called corner cases — that is, under difficult, unusual circumstances — to further sharpen training. Feeding these results into the training pipeline enables neural networks to improve accuracy based on both real and simulated data.
Multi-Robot Sim Is Here
The new SDK provides multi-robot simulation, as well. This allows developers to put multiple robots into a simulation environment for testing, so they learn to work in relation to each other. Individual robots can run independent versions of Isaac’s navigation software stack while moving within a shared virtual environment.
Thus, manufacturers seeking to run multiple robots in large logistics operations can, for example, test their interactions and debug problems before deployment into the real world.
Isaac Integrates with DeepStream
The new SDK also integrates support for NVIDIA DeepStream software, which is widely used for processing analytic capabilities. Video streams can be processed deploying DeepStream and NVIDIA GPUs for AI at the edge supporting robotic applications.
Developers can now build a wide variety of robots that require analysis of camera video feeds, provided for both onboard applications and remote locations.
Programming with Isaac SDK
Finally, for robot developers who have developed their own code, the new SDK is designed to integrate that work, with the addition of a new API based on the C programming language.
This enables developers to connect their own software stacks to the Isaac SDK and minimize programming language conversions — giving users the features of Isaac routed through the C API access.
The inclusion of C-API access also allows the use of the Isaac SDK in other programming languages.
The NVIDIA Isaac SDK 2019.03 download is now available.
Top image: NVIDIA’s Carter robot in the Isaac Sim environment (left) and in the real environment (right).