By Bobby Carlton
NVIDIA’s cuOpt is a game changer executives looking to increase their investments in automation and digital technologies to improve their organizations’ efficiency.
During CES 2023, Nvidia unveiled some incredible new features for its Isaac Sim software that will allow researchers and developers to better train and improve AI robots for various tasks that include areas such as manufacturing, agriculture, retail, and more.
According to Nvidia, the development of AI-based robots requires that they be placed in realistic environments. With the latest version of Isaac Sim, which is now available, developers can now test their models across different operating conditions.
The company’s Isaac platform is composed of various tools such as the ROS module, which runs on the robots and the cuOpt software for route optimization. It also includes the Sim-ready assets, a toolkit for training models, and the TAO optimization system.
“The Isaac robotics platform is designed to accelerate the development and deployment of all manner of robots, and we have a number of software tools and SDKs that address different parts of this solution,” said Gerard Andrews, product manager for Nvidia’s robotics platform, during a CES briefing.
NVIDIA’s tools are built on the foundation of its AI suite and Omniverse, which is a platform that enables the creation and operation of digital twinning applications.
New features
These include new tools and assets for logistics and warehouse operations, such as a conveyor belt utility and a behavior simulation tool for testing safety systems. It additionally has a variety of research tools, such as the Isaac Gym and the Isaac Cortex.
The company’s goal is to provide researchers and developers with the necessary tools and resources to improve and develop AI models for various tasks. According to Andrews, the use of simulation will allow them to create a virtual proof of their creations.
Despite the company’s efforts in simulation, Andrews noted that the company’s work still remains to be done. Some of the factors that will be contributing to the development of new tools include improving the capabilities of its existing tools, such as Isaac Sim, and creating new ones that are specifically designed for different tasks.
“Closing the sim2real gap means the more that the robot performs in simulation like it’s expected to perform in the real world then you are going to get more use cases, more utility, and more value, so we spent a lot of time focusing on how to make our simulations more realistic for that robot user or robot developer,” said Andrews.
He noted that the company also focused on making its tools more flexible and modular. These factors allow the company to provide researchers and developers with the necessary tools and resources to improve and develop AI models for various tasks.
In a recent article on the Nvidia blog site Senior PMM at NVIDIA Robotics, Erin Rapacki wrote about how companies can optimize robot route planning using NVIDIA cuOpt for Isaac Sim.
In Rapacki’s article, she looks at how cuOpt API from NVIDIA enables operations researchers to create real-time fleet routing. It can be used to solve various routing problems, such as job scheduling, robotic route planning, and dynamic rerouting.
The extension for the Isaac Sim simulation environment from NVIDIA includes the cuOpt engine. This component is integrated with the company’s Omniverse application.
“Mailroom workers pick up mail and parcels from different stations and deliver them to various recipients. They know that some envelopes are time-sensitive so they use their knowledge to plan routes with the shortest possible delivery time.
This mail delivery puzzle can be mathematically addressed by using techniques from operations research, a discipline that deals with applying analytical models to improve decision-making and system efficiency. The mathematical science behind operations research is also highly applicable to the process modeling and management of robotics, industrial automation, and material handling systems.”
For logistics professionals, real-time optimization problems are often encountered, such as the travel salesman issue (TSP), vehicle routing problem (VRP, and pickup and delivery problem (PDP).
The more academic version of the travel salesman problem is known as the PDP and VRP. It involves asking a question about the shortest route that can take between each of the destinations, “given a list of destinations and distances between each pair of destinations, what is the shortest possible route that visits each destination exactly once and returns to the original location?”
The use of the travel salesman problem in logistics applications can help reduce the time it takes to move materials from one place to another. For instance, it can be used to improve the efficiency of a manufacturing facility’s transportation network.
In addition, robotics companies can use cuOpt in their planning processes for the deployment of their robots and continuous operation. For instance, during the planning phase of a project, the facility’s process layout can help predict the throughput requirements. This process helps with a successful project ROI, according to the author.
The extension for Isaac Sim from NVIDIA allows continuous operation of the robot fleet while it’s inside the facility. It can be used to route the vehicles according to various system variables, such as the traffic conditions, obstacles, and peak demand for throughput.
Before, companies used a lower-fidelity simulation called discrete event simulator, to design their routing and material handling processes. With the help of cuOpt, they can now use a real-time solution for the planning and implementation of their robots. This component can be used to solve various routing problems, such as the transportation of vehicles and the scheduling of jobs.
McKinsey stated that executives are increasing their investments in automation and digital technologies to improve their organizations’ efficiency. “More than 60 percent of our respondents reported that they have either implemented or are scaling up digital and automation solutions.”
For instance, if a company builds mobile robots or robotic forklifts, it can model how they can move material with varying timeliness compared to people or conveyor belts. To fully understand the system’s systemic differences, it’s important to analyze the entire movement of an object from its origin location to its destination.
To transform existing processes into robotic operations, a company can use the cuOpt extension for Isaac Sim. This component can be utilized to analyze the various steps involved in the design and implementation of their robots, and improve their efficiency, which is outlined below by Rapacki in her article on optimizing robot route planning with cuOpt for Isaac Sim.
Robot readiness:
- Model an existing facility with NVIDIA Isaac Sim.
- Test material movement flow and its throughput requirements.
Redesign of brownfield facilities:
- Model updated facility plans and routing.
- Test updated layouts with simulated robots.
Real-time analytics and rerouting:
- Measure real-time system performance.
- Enable robots to reroute according to process variation.
To help us understand how this works, Rapacki gives us two examples. One in manufacturing and the other in warehousing.
Manufacturing
A manufacturing process involves the timely delivery of parts to the downstream steps of a facility. If the parts arrive late, the factory might not be able to produce as many products that day.
Getting the materials to their destination quickly is a critical component of a manufacturing process, and inefficient route planning can lead to delays.
Warehousing
In warehouses, traffic and floor obstacles can delay the movement of mobile robots. They need dynamic rerouting to react to variable conditions, such as when the route is obscured. If the robots get stuck or slow down, they can act as a constraint or bottleneck and affect the entire operation.
The continuous movement of a material is a critical component of a company’s operations, and it’s important that the robots are always working in the right context. Having the necessary data streams can help floor managers improve the efficiency of their operations.
With the cuOpt extension, a company can easily implement a variety of optimization techniques and improve the efficiency of its operations. It’s built on a patent-pending engine that can evaluate and analyze multiple solutions.
The ability to connect to the performance of NVIDIA’s hardware is a key component of the cuOpt extension. With the ability to create thousands of configurations and environments in a short time, a company can easily improve the efficiency of its processes.
The ability to customize system parameters such as speed of delivery, budget, and robustness can help a company identify the optimal layout for its operations. For instance, in the warehouse and material handling industry, there are specific needs for efficiency and optimization.
For example:
- Improve pick path optimizations
- Increase warehouse capacity
- Improve safety and working conditions
- Maximize storage density
- Improve order fulfillment quality rates
- Plan for multi-flow material movement
- Respond to daily changes in material inputs and bottlenecks
One of the most critical factors that a company can consider when it comes to optimizing its operations is the right operational decisions. With the ability to make dynamic decisions, a company can improve its processes and maximize its output. Through the cuOpt extension, users and robotic companies can benefit from the ability to take action immediately.
This will have a significant impact on the work we do here at FS Studio. For example here is a list of tools we’ve used with current and past projects. Future digital twin projects will absolutely take advantage of the cuOpt extension.
- Development and implementation of Object Detection and Segmentation networks.
- Dataset creation and management using known tools such as CVAT and FiftyOne
- Path and Trajectory planning for robotic manipulators
- Deep learning-based Robotic Grasping algorithms using Point Cloud, Voxels, RGB Images and Depth Images
- Robotic Simulators such as Isaac Sim (NVIDIA), Isaac GYM (NVIDIA), ZeroSim (Unity), Gazebo, Webots
- ROS1 and ROS2 developer
- AI Frameworks: PyTorch and TensorFlow
- Programming languages: Python and C++
NVIDIA’s goal is to make its tools more modular and flexible, and focus on making its simulations more realistic for both developers and researchers. As the number of robots deployed on the market continues to increase, the company’s efforts will continue to be focused on making its tools more capable of handling the challenges of these new robots.
One of the main factors that contributed to the development of the company’s simulation tools is the need to include people in their simulations as workers increasingly interact with robots. This capability allows people to perform certain tasks, such as pushing carts or stacking packages.
“We’re excited about people simulation – the ability to drop characters into the environment and issue commands to those characters and let them take part in a complex event-driven simulation where you can test the software on the robots,” said Andrews.
In the company’s initial release, the tools have a variety of predefined behaviors that allow people to perform certain tasks, such as going to a certain location and avoiding obstacles.
One of the most important factors that the company considered when it came to developing its simulation tools was the need to make them more accurate when it comes to rendering data from sensors. Through the use of NVIDIA RTX technology, the company was able to provide its Isaac Sim with a physically accurate representation of the data collected by the sensors.
“We improved our sensor performance, and specifically for LiDAR, we have ray tracing, which provides accurate performance where the sensor data generated in the simulator starts to mimic and mirror what you’ll get from the real-world sensor.”
According to NVIDIA, ray tracing can provide a more accurate representation of the sensor data in various lighting conditions. It can also support rotating and solid state configurations. Several new models for LiDAR, such as Slamtec, Ouster, and Hesai have been added.
The company’s latest release of its simulation tools includes new 3D assets that can be used to build physically accurate environments. These assets can help speed up the process of creating complex simulations.
The latest version of Isaac Sim also comes with new features for researchers working on complex robot programming and reinforcement learning. These include the Isaac Gym and the Isaac Cortex. A new tool called Isaac ORBIT allows researchers to create functional simulation environments for motion planning and robot learning.
Developers of robot operating systems can now use Isaac Sim’s upgrades for Windows and ROS 2. According to NVIDIA, this will allow them to create complex simulations of the software.
Cloud access
NVIDIA’s focus on the cloud has grown as it allows users to access the latest version of its software and its applications more easily. Andrews noted that this allows the company to benefit from the scalability and accessibility of the cloud.
The availability of Isaac Sim in the cloud allows researchers working on robotic projects to collaborate more easily, and it can help them train and test virtual robots faster. Developers of Isaac replicator software can now create large datasets that can be used to create simulations of real-world environments. They can then use the company’s platform to implement route planning and fleet task management.
The company’s product, known as replicator, is built on the Omniverse technology platform and can be used to create synthetic data models. According to Andrews, it can help researchers train AI models by providing them with a way to supplement their existing data sets.
“We believe simulation is the critical technology to advance robotics and it will be the proving ground for robots,” said Andrews. “We have numerous customers that are working with us that have shared how they have been able to use Isaac Sim so far.”
According to NVIDIA, over a thousand companies and over a million developers have used various parts of the Isaac ecosystem to develop and test virtual robots. Some of these include companies that have used Isaac Sim to develop physical robots.
Use case examples range from Telexistence’s beverage restocking robots and Sarcos Robotics’ robots that pick and place solar panels in renewable energy installations to Fraunhofer’s development of advanced AMRs and Flexiv’s use of Isaac Replicator for synthetic data generation to train AI models.
To begin using the NVIDIA cuOpt for Isaac Sim extension, use the following resources:
- Download NVIDIA Isaac Sim
- See the NVIDIA cuOpt for Isaac Sim demo GitHub repository
- Ramp up your skills with a DLI course, Introduction to Robotic Simulations with Isaac Sim
