AI is moving beyond chatbots and copilots into the physical world. Across laboratories, factories and hospitals, a new generation of AI agents is beginning to work alongside people, helping them understand their environment, access knowledge and take action in real time.
However, building agentic systems that combine models, skills, harnesses, tools and an agentic runtime to help people perform hands-on work is challenging. To operate effectively in dynamic, real-world environments, these agents must do more than generate responses.
Like human workers, they need knowledge, tools and specialized skills to perceive and understand the world through video, audio and sensor data, interpret fast-changing conditions and spatial context, retrieve information from enterprise systems, reason about the next best action and use software tools to complete tasks. All of this must happen with low latency and in a way that supports the user without creating distraction.
NVIDIA XR AI is a developer library that helps developers build these agentic applications. By connecting inputs from AR glasses and XR devices with AI models, enterprise data, tools and accelerated computing, NVIDIA XR AI enables agents that can perceive, reason and act in the flow of work.
It provides a foundation for developers to build or connect skills and tools for enterprise XR applications, and simplifies the integration of multimodal perception, enterprise retrieval, reasoning models and agent orchestration. Together, these capabilities make it easier to build spatially aware, multimodal AI agents that deliver low-latency, context-aware assistance in AR and XR experiences.
The platform brings together four core capabilities:
- Ingests real-world signals from AR and XR devices, including video, audio, depth, pose and sensor data.
- Connects agents to specialized tools and services, including NVIDIA Metropolis and the NVIDIA Metropolis for video search and summarization (VSS) for visual AI and video understanding, and NVIDIA NeMo Retriever for enterprise knowledge retrieval and retrieval-augmented generation.
- Supports a broad ecosystem of AI models, including NVIDIA Nemotron reasoning models, NVIDIA Cosmos Reason and other compatible foundation models.
- Integrates agent orchestration and accelerated runtime services to help developers move from prototype to production.
NVIDIA NeMo Agent Toolkit enables tool use, reasoning workflows and multi-agent coordination, while NVIDIA accelerated computing platforms — including NVIDIA DGX Spark, NVIDIA DGX Station and NVIDIA RTX PRO systems — provide the infrastructure to run inference across cloud, data center and edge environments.
Together, these capabilities enable AI agents that can understand their surroundings, access enterprise knowledge, reason about complex tasks and deliver contextual assistance in real time.
Industries Put NVIDIA XR AI to Work
Across manufacturing, science, healthcare, design and immersive learning, developers and enterprises are already tapping NVIDIA XR AI — embedding AI agents where the work happens.
Siemens is exploring in a research context how NVIDIA XR AI and NVIDIA DGX Spark can help factory engineers find maintenance information, troubleshoot issues, verify work and capture what happened on the shop floor.
With this system, an engineer wearing lightweight glasses can ask an AI agent about a programmable logic controller issue and receive real-time guidance, connecting industrial systems, digital twins and automation workflows.
In the research lab, Rana, an AutoBio company building AI systems for scientific research, is introducing its LabOS system on NVIDIA XR AI to bring spatial intelligence directly into scientific workflows. LabOS provides real-time, hands-free guidance for complex experimental workflows, starting with stem cell therapy and gene-editing research at the Cong Lab at Stanford University School of Medicine and the Wang Lab at Princeton University.
Built on the XR AI architecture, the LabOS co-scientist perceives, understands and acts within the lab environment, helping researchers identify the right sample and CRISPR gene editor, guiding each experimental step and capturing a structured, reproducible record as humans, robots and AI systems collaborate at the bench.
Physically aware AI agents, delivered through AR glasses and powered by NVIDIA GPUs, serve as a next-generation interface for AI-assisted science — keeping researchers focused on complex procedures while receiving contextual guidance in real time.
LabOS is compatible with smart glasses from Meta, Rokid and VITURE.
VITURE integrated NVIDIA XR AI into a wearable interface that gives workers a hands-free way to find the right context and guide the next step at the point of work. This same XR AI foundation extends naturally beyond the lab, into clinics and industrial settings.
In the operating room, the Surreality Lab at University of Pittsburgh Medical Center showcased how NVIDIA XR AI can support surgical teams with context-aware assistance. Running on NVIDIA XR AI and NVIDIA DGX Station, the pipeline is designed to help teams find information and guide attention without adding visual clutter for the surgeon.
By understanding what not to occlude in the surgeon’s view, the system can surface useful context while preserving focus on the patient and procedure.
In automotive design, Innoactive shows how enterprises can capture relevant information and data during immersive workflows to support design decision-making.
Powered by an NVIDIA DGX Spark system, the experience helps teams preserve context from design reviews, product showrooms and digital twins so spatial work can move from one-off sessions to repeatable enterprise processes.
Atlantic Studios, a multi-Academy- and Emmy-winning storytelling and immersive media studio, is using NVIDIA XR AI to let audiences explore an immersive scan of the Titanic as it rests today.
Users can use voice prompts to find points of interest and guide discovery through the historic site — turning a complex underwater model into an interactive spatial story that answers questions, surfaces context and helps users learn in real time.
As AI agents gain the ability to perceive the physical world, use tools, access enterprise knowledge and collaborate with people, they are becoming a new class of digital workers. NVIDIA XR AI provides the libraries and accelerated computing foundation developers need to build these agents for laboratories, factories, hospitals and immersive environments — bringing agentic AI directly into the flow of work.
Learn more about NVIDIA XR AI and access the developer resources.
See notice regarding software product information.
Categories:
Pro Graphics
Related News
