The gap between reality and simulation just became narrower.
In his keynote at GTC, NVIDIA founder and CEO Jensen Huang announced NVIDIA Omniverse Replicator, an engine for generating synthetic data with ground truth for training AI networks. In a demo, Huang showed the power of Omniverse Replicator when applied to autonomous vehicle development using DRIVE Sim.
DRIVE Sim is a simulation tool built on Omniverse that takes advantage of the many capabilities of the platform. Data generated by DRIVE Sim is used to train deep neural networks that make up the perception systems in autonomous vehicles. For the NVIDIA DRIVE team, synthetic data has been an effective, critical component of its AV development workflow.
The deep neural networks powering an autonomous vehicle’s perception are composed of two parts: an algorithmic model, and the data used to train that model. Engineers have dedicated significant time to refining algorithms. However, the data side of the equation is still underdeveloped due to the limitations of real-world data, which is incomplete and time consuming and costly to collect.
This imbalance often leads to a plateau in DNN development, hindering progress where the data cannot meet the demands of the model. With synthetic data generation, developers have more control over data development, tailoring it to the specific needs of the model.
While real-world data is a critical component for AV training, testing and validation, it presents significant challenges. The data used to train these networks is collected by sensors on a vehicle fleet during real-world drives. Once captured, the data must be labeled with the ground truth. Annotation is done by hand by thousands of labelers — a process that is time consuming, costly and can be inaccurate.
Augmenting real-world data collection with synthetic data removes these bottlenecks while allowing engineers to take a data-driven approach to DNN development, significantly accelerating AV development and improving real-world results.
The Sim-Real Domain Gap Problem
Synthetic data generation is a well-known tool for AI training — researchers have been experimenting with video games such as Grand Theft Auto to create data as early as 2016.
Unlike video games, however, the quality of perception DNNs is heavily influenced by the fidelity of the data to the real world — training on datasets that don’t translate to the physical world can actually degrade a network’s performance.
This sim-to-real gap presents itself in two ways primarily. An appearance gap corresponds to pixel-level differences between the simulated image and the real image, which is caused by how the simulator generates the data. The renderer, sensor model, fidelity of 3D assets and material properties can all contribute to this.
A content gap can be caused by the lack of real-world content diversity and by differences between sim and real-world contexts. These inconsistencies occur when the context of a scene doesn’t match reality. For example, the real world contains dirty roads, dented cars, and emergency vehicles on roadsides, which all must be reproduced in simulation. Another important factor is the behavior of actors, such as traffic and pedestrians — realistic interactions are key to realistic data output.
Applying Omniverse Replicator to Narrow the Sim-Real Domain Gap
Omniverse Replicator is designed to narrow the appearance and content gaps.
To narrow the appearance gap, DRIVE Sim takes advantage of Omniverse’s RTX path-tracing renderer to generate physically based sensor data for cameras, radars and lidars and ultrasonics sensors. Real-world effects are captured in the sensor data, including phenomena such as LED flicker, motion blur, rolling shutter, lidar beam divergence and doppler effect. These details even include high-fidelity vehicle dynamics, which are important because, for example, the movement of a vehicle during a lidar scan impacts the resulting point cloud.
The other half of this sensor equation is materials. Materials in DRIVE Sim are physically simulated for accurate beam reflections. DRIVE Sim includes a built-in lidar materials library and a forthcoming radar and ultrasonics materials library.
One of the primary ways to address the content gap is with more diverse assets at the highest levels of fidelity. DRIVE Sim leverages Omniverse’s capabilities to connect to a wide range of content creation tools. However, generating proper scenes also requires that the context is correct.
Under the hood, Omniverse Replicator organizes data for rapid scene manipulation using a capability called domain randomization. DRIVE Sim includes tools for this and scene construction that create a large amount of diverse data while maintaining real-world context. Since Omniverse Replicator is also time accurate and deterministic, the dataset can be created in repeatable way.
DRIVE Sim provides tools for generating randomized scenes in a controlled and repeatable way that adds variety and diversity to the data generated.
Real Results
DRIVE Sim has already produced significant results in accelerating perception development with synthetic data at NVIDIA.
One example is the migration to the latest NVIDIA DRIVE Hyperion sensor set. The NVIDIA DRIVE Hyperion 8 platform includes sensors for complete production AV development. However, before these sensors were even available, the NVIDIA DRIVE team was able to bring up DNNs for the platform using synthetic data. DRIVE Sim generated millions of images and ground-truth data for training. As a result, the networks were ready to deploy as soon as the sensors were installed, saving valuable months of development time.
In another instance, the PathNet DNN, which detects drivable lane space, had difficulty determining a path when the vehicle was not centered in the lane. Collecting such data is difficult because driving partially out of a lane is dangerous (and against NVIDIA’s data collection policy). By training the network on millions of synthetic images of off-centered driving paths, DRIVE Sim significantly improved PathNet’s accuracy.
The same proved true for both LightNet, which detects traffic lights, and SignNet, which detects and classifies road signs. These networks were struggling to identify lights at extreme angles and misclassifying signs in certain conditions due to a lack of data. Engineers were able to design data to augment the real-world datasets and improve performance.
By training both DNNs on synthetic data that covered these problem areas, performance quickly improved, removing bottlenecks from the development process.
Seeing What Humans Can’t
Synthetic data changes the nature of DNN development. It’s time- and cost-effective, and provides engineers the freedom to generate a tailored dataset on demand.
Developers can specify elements such as weather, lighting, pedestrians, road debris, and more. They can also control the distribution of elements, such as specifying a particular mix of trucks, busses, cars and motorcycles in a given set of data.
Synthetic data provides ground truth that humans can’t label. Examples are depth information, velocity and multisensor tracking. This ground truth information can significantly enhance perception capabilities.
It also facilitates labeling for components that are difficult, sometimes impossible, to label. For example, a pedestrian that walks behind a car is impossible for a human to label properly while occluded. However with simulation, the ground truth is available automatically and accurate at a pixel level, even if the information isn’t visible to humans.
A key feature of synthetic data is precise ground-truth labels for scenes that are difficult or impossible in the real world, such as this scene with the pedestrian who is occluded as the car passes.
Clearing the Path Ahead
As a modular, open and extensible platform for synthetic data generation, Omniverse Replicator brings powerful new capabilities to deep-learning engineers. DRIVE Sim uses these new features to provide AV developers the ultimate flexibility and efficiency in simulation testing.
It allows engineers to create the datasets they need to accelerate their work.
The results are DNNs that are more accurate and developed on a shorter timeline, bringing safer, more efficient autonomous driving technology to roads sooner.
Learn more about DRIVE Sim and accelerate the development of safer, more efficient transportation today.
Catch up on the full GTC keynote: