Summary
In this article, the team at FS Studio shares their ideas on increasing the speed and efficacy of training and implementing robots to increase workplace efficiency and productivity. This article explores possible solutions like an AR application using the Magic Leap to demonstrate the potential for using AR as a platform for setting up, training, and working with workplace robots.
Problems
Workplace robots are undoubtedly the future of increasing organizational efficiency and productivity; however, there are several problems that developers and managers face when it comes to training these robots at scale and acclimating robots to their workplace environment.
Most of the problems arise because robots are inherently spatial. They move, their arms can change shape and position, and they are generally designed to manipulate or move things in different ways. Current computing methods are not inherently spatial and we struggle to adapt them to the unique problems of giving instructions to machines that include inherently spatial aspects.
PROBLEM 1. ROBOTS NEED TO BE TRAINED IN THE ENVIRONMENT WHERE THEY WILL BE WORKING
3D environments are a commonplace way of training robots, and one of the significant cost and labor factors is modeling the 3D environment to match the real environment the robot will be used in.
PROBLEM 2. MODELING THE ENVIRONMENT WHERE THE ROBOT WILL WORK IS EXPENSIVE & LABOR INTENSIVE
Often, this stage is left until after delivery of a robot and accompanies a lengthy setup period where the robot is unproductive, and the workspace is also tied up as the robot is trained in place.
PROBLEM 3. THE TEMPORAL CLIFF BETWEEN TRAINING THE ROBOT IN THE WORKPLACE ENVIRONMENT TEMPORARILY REDUCES ORGANIZATIONAL EFFICIENCY AND PRODUCTIVITY.
Even after the robot is programmed and trained, it takes a long period of time to acclimate the robot to its workplace environment. In-place training of the robot also requires the robot to be set up in the work environment, which means the robot is not productive. This also means that the workplace is also not able to be productive until training is done.
PROBLEM 4: IT IS DIFFICULT USING CURRENT TECHNOLOGY FOR HUMANS TO LEARN HOW TO TRAIN ROBOTS TO PERFORM COMPLEX MANIPULATION TASK
Possible Solutions & Our Full Vision
Our main premise is that sufficiently sophisticated AR technology is well suited to interacting with and controlling robots because it is inherently spatial. Spatial computing for AR is therefore inherently well suited to working with spatial machinery.
Imagine ordering an industrial robot, and receiving an AR headset with a pre-installed app designed for your robot. You put the headset on, and it guides you through the process of evaluating your robot environment, identifying safety issues, laying out where the robot will work and where it can’t go, and even training a virtual version of the robot on specific jobs it will do, all without even having the real robot uncrated.
Even once the real robot is in use, the operator would still use the AR headset to give instructions to the robot. Instructions given to robots are usually complex and are once again spatial in nature. Advanced robots also use visual SLAM techniques for machine vision, so there is a good match with how AR works.
Being able to simply look at things, point at them, pick them up and put them down and have the AR headset see and interpret what you are doing relative to the environment goes a long way toward allowing for a system that can then instruct the robot to do the same kind of thing.
Trying to describe this kind of interaction using flat screens, mice and keyboards is frustrating, error prone, and non intuitive. AR headsets bridge the world between robotics and humans like no other technology can do.
SOLUTION 1. AR TECHNOLOGIES INHERENTLY CREATE A 3D MODEL OF THE SPACE WHERE THE ROBOT WILL WORK
We postulate that AR technologies – such as the Magic Leap or Hololens – have the potential to streamline the entire process of training and implementing workplace robots. These technologies make it possible for an organization to do a lot of prep work way before the robot even arrives on the site. The business imperative behind this is that it decreases the amount of time that a workplace area will be closed off for robot training – for some organizations, even a few days of downtime can be a make or break.
This technology could allow the operator to walk around the work environment wearing an advanced AR headset, and a 3D mesh corresponding to the environment will be created in real time and visualized in the headset, and the operator can set up robot workflows and other tasks directly in the AR headset.
For example, the operator can point at various locations on the floor or walls to identify path markers or workstation locations, and a virtual version of the robot can then be deployed in AR to follow the path markers or positioned to set up manipulation tasks at the workstations.
This method eliminates the need for a 3D environment to be created by a 3D artist, and this would enhance the accuracy of the 3D model since it will be updated in real time on location. This first step in the process would decrease the workplace downtime and increase the efficacy of training the robot.
SOLUTION 2. AR TECHNOLOGY CAN INCREASE THE SAFETY AND EFFICACY OF HUMAN-ROBOT INTERACTIONS
Robots and workers are increasingly used together in pairs, so it’s important to have tools to keep them from coming into conflict. It is already common practice to identify areas within a robotic workspace where robots should never go, but the tools to mark out such areas work on an abstract 3D model of the workplace and don’t allow good visualization of the result.
This technology would allow workers to highlight such areas simply by looking directly at the area in the AR headset and directly drawing the no go zone on the floor (or on the ceiling).
With the hand and finger tracking features of Magic Leap, it is even possible to directly manipulate the robot arm with just the operators hand motions using IK to position the robot, in a similar way to how current robots like Baxter allow training with a real robot. Direct positioning of the robot arm is one of the most intuitive ways for humans to train robots.
SOLUTION 3. AR TECHNOLOGY CAN INCREASE THE SCALABILITY AND EFFECTIVENESS OF ROBOT-EMPLOYEE TRAINING
It should also be possible to transfer the mesh and other information from an AR app into a VR app for further refinement and training by workers who are not on site. Using a cloud based approach to storing the environment model, a mix of AR and VR workers can be supported without regard to if they are using VR or AR.
SOLUTION 4. AR TECHNOLOGY COULD SHORTEN THE AMOUNT OF TIME IT TAKES SET UP A ROBOT IN A WORKPLACE ENVIRONMENT
Other features like image recognition, gaze tracking and voice recognition could also make setting up a robotic workstation much simpler and more intuitive by letting an operator work with a fully functional virtual version of the robot, and then transferring the training to the real robot later once it arrives.
The result would be much less disruption of the workplace the robot is being introduced to, and a much quicker time to the robot being productive in its new location.
Deconstructing Current Implementation
The current proof of concept is implemented using Magic Leap ML1 headset and the Unity game engine. Conceptually, it consists of three main objects – The world, the robot, and the user.
The World
The app is based on one of the Magic Leap examples that show how ‘meshing’ works. Meshing is the process of using the front facing cameras on the ML1 and stereo photogrammetry to create a 3D representation or ‘mesh’ of the users environment that virtual objects can then interact with.
This is ideal for the purposes of a robotic simulator since the robots we want to simulate have their own sensors for detecting features in the real world environment and we can simulate these sensors using the mesh we’re getting from the Magic Leap AR platform.
The Robot
A Fetch robotics robot is created in the AR environment using the URDF format and textures and models from the Fetch robotics website.
The textures were modified slightly to avoid using very dark colors since these tend to become transparent in AR because it uses additive display technology to allow seeing the real world through the AR headset.
The robot Physics model uses the parameters for mass, damping coefficients, maximum speeds, etc. from the URDF definition to create PID control loops and simulations for each of the robot limbs so they move at realistic rates in the simulation. The robot includes a sophisticated state machine to allow it to idle, move, be manipulated using IK and other functions as needed.
The Robot includes a pair of machine vision cameras on the robot ‘head’ as well as a laser range finder aimed with the head, and a LIDAR based range finder in the base that scans a 120 degree arc in front of the robot. In our AR simulations, these laser range finders actually ‘work’ and allow the virtual robot to detect the real world geometry, making it possible to teach navigation based tasks using the 3D data for the actual environment.
Using a third party SDK for Unity called Final IK, it is possible to ‘grab’ the end effector using the Magic Leap controller, and position it arbitrarily in space with the arm following using inverse kinematics. IK is still very coarse and the model doesn’t check for self intersection or collisions with the environment.
Our robot also includes a UX panel floating above it that is used to view the status of the robot, and to initiate or play back sequences of motions it has been trained for.
The User
The user has two main components. There is the headset and associated children in the hierarchy, and the controller they use to point with or grab things. There are also a few other objects in the AR scene used to display help.
The Headset
The headset tracks the movements of the ML1 as the user moves around in the environment, the ML1 is a 6DOF device so position as well as rotation is tracked. The headset contains the cameras used for stereo photogrammetry, so environment meshing takes place in a cone in front of the user and they need to move around and look in different directions to fill in the AR mesh.
The headset also contains speakers, and we use positional audio so sounds come from the objects they should come from such as the controller or the robot.
The Controller
The controller is used like a standard laser pointer to select and manipulate things that are far away, and as a spatial reference for determining proximity to the end effector to initiate IK positioning.
When used as a laser pointer, the user can point at and interact with the robot UX control that contains familiar 2D UI elements like buttons.
The can also point at any of the joints on the robot, and use the thumbpad to rotate the joint directly using forward kinematics.
We use the haptics features of the controller to provide feedback to users.
Next Steps
NEXT STEP 1. REWORK THE INVERSE KINEMATICS
The most important thing would be to rework the inverse kinematics the robot is using so that the robot arm would never self intersect with other parts of the robot.
NEXT STEP 2. IMPLEMENT PATHFINDING AND WAYPOINT NAVIGATION
Another important feature would be to use the simulated LIDAR data to implement pathfinding and waypoint navigation that takes the real world mesh into account and avoids collisions.
NEXT STEP 3. EXPLORE POSSIBLE SOLUTIONS TO CONTROLLER-LESS IK
Allowing the user to do IK on the robot arm using their hands only without the controller would be very desirable.
Conclusion
A lot of work will need to be done to turn our proof of concept into a viable robot training app; however, we unequivocally believe that this methodology will increase efficacy, safety and productivity and will also reduce the time it takes to train and set up a workplace robot.
