FS Studio Logo

Recently, the leading B2B ratings and reviews company Clutch announced the most highly-rated software development services companies in 5 major cities in the U.S., including San Francisco, CA.

We’re excited to announce that our team at FS Studio was highlighted as a leading software developer based in SF in 4 major categories! :

Emerging technology and advanced software design is our specialty.  If there’s a tough technical nut to crack, that’s what we do best. If you’re looking to create an extra special AR/VR experience, we have done it.

Clutch’s analysts speak to each company’s client references about the business challenges, solutions and results during their time working together. The research firm uses a scoring methodology with factors including clients, market presence, and industry recognitions, strengthening their evaluation. The platform then forms a Leaders Matrix for a particular industry segment and identifies firms with the ability to deliver.

Assessing our ability in software development, Clutch did their assessment and round-up of client feedback. Here are some comments from our client reviews on our Clutch profile:

Custom Development For Boutique Marketing Agency

“FS Studio has deeper technical chops than other development shops, so we trust them when it comes to front - and back-end development. We think their ability to scale resources according to project requirements is remarkable for a development agency that works with creative agencies. A lot of the other development agencies have far less bandwidth.”

 

Platform Development for LeapFrog Enterprises

“The fact that LeapTV made it into the marketplace last year, with all of the functionality that we had wanted it to have, is a testament to not only their raw development ability but also the teamwork and collaboration that they bring to their projects, the level of communication that they bring to their projects.”

We’re really proud of our team for this recognition by Clutch and the fact that we were featured in 4 segments. We look forward to continuing our success and being recognized again next year!

Summary

In this article, the team at FS Studio shares their ideas on increasing the speed and efficacy of training and implementing robots to increase workplace efficiency and productivity. This article explores possible solutions like an AR application using the Magic Leap to demonstrate the potential for using AR as a platform for setting up, training, and working with workplace robots.

Problems

Workplace robots are undoubtedly the future of increasing organizational efficiency and productivity; however, there are several problems that developers and managers face when it comes to training these robots at scale and acclimating robots to their workplace environment.

Most of the problems arise because robots are inherently spatial. They move, their arms can change shape and position, and they are generally designed to manipulate or move things in different ways. Current computing methods are not inherently spatial and we struggle to adapt them to the unique problems of giving instructions to machines that include inherently spatial aspects.

PROBLEM 1. ROBOTS NEED TO BE TRAINED IN THE ENVIRONMENT WHERE THEY WILL BE WORKING

3D environments are a commonplace way of training robots, and one of the significant cost and labor factors is modeling the 3D environment to match the real environment the robot will be used in.

PROBLEM 2. MODELING THE ENVIRONMENT WHERE THE ROBOT WILL WORK IS EXPENSIVE & LABOR INTENSIVE

Often, this stage is left until after delivery of a robot and accompanies a lengthy setup period where the robot is unproductive, and the workspace is also tied up as the robot is trained in place.

PROBLEM 3. THE TEMPORAL CLIFF BETWEEN TRAINING THE ROBOT IN THE WORKPLACE ENVIRONMENT TEMPORARILY REDUCES ORGANIZATIONAL EFFICIENCY AND PRODUCTIVITY.

Even after the robot is programmed and trained, it takes a long period of time to acclimate the robot to its workplace environment. In-place training of the robot also requires the robot to be set up in the work environment, which means the robot is not productive. This also means that the workplace is also not able to be productive until training is done.

PROBLEM 4: IT IS DIFFICULT USING CURRENT TECHNOLOGY FOR HUMANS TO LEARN HOW TO TRAIN ROBOTS TO PERFORM COMPLEX MANIPULATION TASK

Possible Solutions & Our Full Vision

Our main premise is that sufficiently sophisticated AR technology is well suited to interacting with and controlling robots because it is inherently spatial. Spatial computing for AR is therefore inherently well suited to working with spatial machinery.

Imagine ordering an industrial robot, and receiving an AR headset with a pre-installed app designed for your robot. You put the headset on, and it guides you through the process of evaluating your robot environment, identifying safety issues, laying out where the robot will work and where it can’t go, and even training a virtual version of the robot on specific jobs it will do, all without even having the real robot uncrated.

Even once the real robot is in use, the operator would still use the AR headset to give instructions to the robot. Instructions given to robots are usually complex and are once again spatial in nature. Advanced robots also use visual SLAM techniques for machine vision, so there is a good match with how AR works.

Being able to simply look at things, point at them, pick them up and put them down and have the AR headset see and interpret what you are doing relative to the environment goes a long way toward allowing for a system that can then instruct the robot to do the same kind of thing.

Trying to describe this kind of interaction using flat screens, mice and keyboards is frustrating, error prone, and non intuitive. AR headsets bridge the world between robotics and humans like no other technology can do.

SOLUTION 1. AR TECHNOLOGIES INHERENTLY CREATE A 3D MODEL OF THE SPACE WHERE THE ROBOT WILL WORK

We postulate that AR technologies - such as the Magic Leap or Hololens - have the potential to streamline the entire process of training and implementing workplace robots. These technologies make it possible for an organization to do a lot of prep work way before the robot even arrives on the site. The business imperative behind this is that it decreases the amount of time that a workplace area will be closed off for robot training - for some organizations, even a few days of downtime can be a make or break.

This technology could allow the operator to walk around the work environment wearing an advanced AR headset, and a 3D mesh corresponding to the environment will be created in real time and visualized in the headset, and the operator can set up robot workflows and other tasks directly in the AR headset.

For example, the operator can point at various locations on the floor or walls to identify path markers or workstation locations, and a virtual version of the robot can then be deployed in AR to follow the path markers or positioned to set up manipulation tasks at the workstations.

This method eliminates the need for a 3D environment to be created by a 3D artist, and this would enhance the accuracy of the 3D model since it will be updated in real time on location. This first step in the process would decrease the workplace downtime and increase the efficacy of training the robot.

SOLUTION 2. AR TECHNOLOGY CAN INCREASE THE SAFETY AND EFFICACY OF HUMAN-ROBOT INTERACTIONS

Robots and workers are increasingly used together in pairs, so it’s important to have tools to keep them from coming into conflict. It is already common practice to identify areas within a robotic workspace where robots should never go, but the tools to mark out such areas work on an abstract 3D model of the workplace and don’t allow good visualization of the result.

This technology would allow workers to highlight such areas simply by looking directly at the area in the AR headset and directly drawing the no go zone on the floor (or on the ceiling).

With the hand and finger tracking features of Magic Leap, it is even possible to directly manipulate the robot arm with just the operators hand motions using IK to position the robot, in a similar way to how current robots like Baxter allow training with a real robot. Direct positioning of the robot arm is one of the most intuitive ways for humans to train robots.

SOLUTION 3. AR TECHNOLOGY CAN INCREASE THE SCALABILITY AND EFFECTIVENESS OF ROBOT-EMPLOYEE TRAINING

It should also be possible to transfer the mesh and other information from an AR app into a VR app for further refinement and training by workers who are not on site. Using a cloud based approach to storing the environment model, a mix of AR and VR workers can be supported without regard to if they are using VR or AR.

SOLUTION 4. AR TECHNOLOGY COULD SHORTEN THE AMOUNT OF TIME IT TAKES SET UP A ROBOT IN A WORKPLACE ENVIRONMENT

Other features like image recognition, gaze tracking and voice recognition could also make setting up a robotic workstation much simpler and more intuitive by letting an operator work with a fully functional virtual version of the robot, and then transferring the training to the real robot later once it arrives.

The result would be much less disruption of the workplace the robot is being introduced to, and a much quicker time to the robot being productive in its new location.

Deconstructing Current Implementation

The current proof of concept is implemented using Magic Leap ML1 headset and the Unity game engine. Conceptually, it consists of three main objects - The world, the robot, and the user.

The World

The app is based on one of the Magic Leap examples that show how ‘meshing’ works. Meshing is the process of using the front facing cameras on the ML1 and stereo photogrammetry to create a 3D representation or ‘mesh’ of the users environment that virtual objects can then interact with.

This is ideal for the purposes of a robotic simulator since the robots we want to simulate have their own sensors for detecting features in the real world environment and we can simulate these sensors using the mesh we’re getting from the Magic Leap AR platform.

The Robot

A Fetch robotics robot is created in the AR environment using the URDF format and textures and models from the Fetch robotics website.

The textures were modified slightly to avoid using very dark colors since these tend to become transparent in AR because it uses additive display technology to allow seeing the real world through the AR headset.

The robot Physics model uses the parameters for mass, damping coefficients, maximum speeds, etc. from the URDF definition to create PID control loops and simulations for each of the robot limbs so they move at realistic rates in the simulation. The robot includes a sophisticated state machine to allow it to idle, move, be manipulated using IK and other functions as needed.

The Robot includes a pair of machine vision cameras on the robot ‘head’ as well as a laser range finder aimed with the head, and a LIDAR based range finder in the base that scans a 120 degree arc in front of the robot. In our AR simulations, these laser range finders actually ‘work’ and allow the virtual robot to detect the real world geometry, making it possible to teach navigation based tasks using the 3D data for the actual environment.

Using a third party SDK for Unity called Final IK, it is possible to ‘grab’ the end effector using the Magic Leap controller, and position it arbitrarily in space with the arm following using inverse kinematics. IK is still very coarse and the model doesn’t check for self intersection or collisions with the environment.

Our robot also includes a UX panel floating above it that is used to view the status of the robot, and to initiate or play back sequences of motions it has been trained for.

The User

The user has two main components. There is the headset and associated children in the hierarchy, and the controller they use to point with or grab things. There are also a few other objects in the AR scene used to display help.

The Headset

The headset tracks the movements of the ML1 as the user moves around in the environment, the ML1 is a 6DOF device so position as well as rotation is tracked. The headset contains the cameras used for stereo photogrammetry, so environment meshing takes place in a cone in front of the user and they need to move around and look in different directions to fill in the AR mesh.

The headset also contains speakers, and we use positional audio so sounds come from the objects they should come from such as the controller or the robot.

The Controller

The controller is used like a standard laser pointer to select and manipulate things that are far away, and as a spatial reference for determining proximity to the end effector to initiate IK positioning.

When used as a laser pointer, the user can point at and interact with the robot UX control that contains familiar 2D UI elements like buttons.

The can also point at any of the joints on the robot, and use the thumbpad to rotate the joint directly using forward kinematics.

We use the haptics features of the controller to provide feedback to users.

Next Steps

NEXT STEP 1. REWORK THE INVERSE KINEMATICS

The most important thing would be to rework the inverse kinematics the robot is using so that the robot arm would never self intersect with other parts of the robot.

NEXT STEP 2. IMPLEMENT PATHFINDING AND WAYPOINT NAVIGATION

Another important feature would be to use the simulated LIDAR data to implement pathfinding and waypoint navigation that takes the real world mesh into account and avoids collisions.

NEXT STEP 3. EXPLORE POSSIBLE SOLUTIONS TO CONTROLLER-LESS IK

Allowing the user to do IK on the robot arm using their hands only without the controller would be very desirable.

Conclusion

A lot of work will need to be done to turn our proof of concept into a viable robot training app; however, we unequivocally believe that this methodology will increase efficacy, safety and productivity and will also reduce the time it takes to train and set up a workplace robot.

Recently, the leading B2B ratings and reviews company Clutch announced the most highly-rated software development services companies in 5 major cities in the U.S., including San Francisco, CA.

We’re excited to announce that our team at FS Studio was highlighted as a leading software developer based in SF in 4 major categories! :

Emerging technology and advanced software design is our speciality.  If there’s a tough technical nut to crack, that’s what we do best. If you’re looking to create an extra special AR/VR experience, we have done it.

Clutch’s analysts speak to each company’s client references about the business challenges, solutions and results during their time working together. The research firm uses a scoring methodology with factors including clients, market presence, and industry recognitions, strengthening their evaluation. The platform then forms a Leaders Matrix for a particular industry segment and identifies firms with the ability to deliver.

Assessing our ability in software development, Clutch did their assessment and round-up of client feedback. Here are some comments from our client reviews on our Clutch profile:

Custom Development For Boutique Marketing Agency

“FS Studio has deeper technical chops than other development shops, so we trust them when it comes to front - and back-end development. We think their ability to scale resources according to project requirements is remarkable for a development agency that works with creative agencies. A lot of the other development agencies have far less bandwidth.”

 

Platform Development for LeapFrog Enterprises

“The fact that LeapTV made it into the marketplace last year, with all of the functionality that we had wanted it to have, is a testament to not only their raw development ability but also the teamwork and collaboration that they bring to their projects, the level of communication that they bring to their projects.”

 

We’re really proud of our team for this recognition by Clutch and the fact that we were featured in 4 segments. We look forward to continuing our success and being recognized again next year!

VR and AR development across multiple platforms has become a big part of our professional portfolio.  We've done VR game development with support development work for integrating specialized peripheral hardware, we've done educational VR titles, and we've done real-time computer vision marker detection and tomography for AR applications.

Being a part of the VR/AW Association is opening up a huge network of collaborators and markets for us and we're proud to be new members!

Our Space Battle opens up a new frontier in the exploration for the most exciting interactive toys

What’s the future of play? How are advances in engineering changing the world of game design and toys? In an age of touch screens, 3D graphics and augmented reality, it goes without saying that our children are already assimilated to the connected world. I recently spoke to a curator for a children’s museum who reiterated the importance of promoting interactivity in design for educational curriculum to keep up with the pace of entertainment and other media that increasingly make fractal claims on our children’s attention.

“If they can’t touch it and have the object respond,” she told me, “as far as they’re concerned, it’s a poster.”

Every day at FS Studio, we try to tackle the challenges of the next generation toy. Every day we play Geppetto to our clients’ Pinnochio, trying to conjure something unique and alive out of a magical block of talking wood (in this case, plastic and silicon chips).

Introducing SPACE BATTLE, a smart toy we’re excited to debut on the Artik platform. When we prototyped SPACE BATTLE, we wanted to enhance a toy that by all appearances looked like a regular spaceship toy but with the latest capabilities in AI. Equipped with an optical sensor, SPACE BATTLE uses computer vision to respond to its environment. We also wanted to introduce an engaging storyline that facilitates participation and wonder. In other words, the toy plays with you.

We wanted SPACE BATTLE to have all the autonomy and portability of a regular toy that a child can carry around anywhere regardless of connectivity. The results of our experiment are a window into the ever expanding world of “enchanted” or smart devices. The evolution from static “dumb” toys to a future of interactive toys, toys that are fully embedded with sophisticated deep learning capabilities and our own proprietary mix of technology and fun that allows them to perform complicated tasks while running completely offline (no Internet required!).

How did we do it?

The challenge, of course, was optimization. For the toy to be feasible, it had to run a real-time image recognition process that works on low-powered devices. That’s where the Artik comes in. Its quadcore processors were more then capable. We were able to equip SPACE BATTLE with a convolutional neural net, or “deep net,” that allowed the rocket ship to come alive in yours hands.

If tech talk makes your eye glaze over, feel free to jump to the end, if you want to get a peek into the magic that made this happen, read on!

Starts with the Dataset

So what’s happening under the proverbial hood, lets start with the dataset creation. We undertook the task of “training” our neural network with tens of thousands of images. This was done with a blend of both real images and a synthetically generated image dataset. The real images are absolutely the best source of data for training the neural net, you get natural lighting, backlighting, shadows, and all the subtleties that you can miss in a synthetically generated dataset. However the synthetic dataset allowed us to create a vast amount of data to augement these real images and this helped train the neural net with offset images, rotated images, varied image sizes, and many many more backgrounds. These synthetic images were created in a 2D image environment, we are looking to creating tools that use OpenGL and 3D environments for synthetic image creation, which get us closer to real images especially when it comes to things like lighting, camera placement, and foreground occlusion.

The key to modern deep learning techniques is the use of Stochastic Gradient Descent (SGD). Then we tweak, tweak, tweak the training hyperparameters, use of mini-batch sizes, number of training iterations, steps sizes, alpha, on and on and on (we also tweak the network similarly).

How do we verify our results of all this tweaking, two ways. We set aside a portion of our training data for verification. This gives us a measure of confidence in our accuracy, but to be sure using the training data and getting good results against those is no guarantee that we will get similar results in the real-world. That’s our final litmus test, getting out there and brute force manual testing in the real world. We get this into the hands of as many folks as we can and see what our results are, this is decidedly qualitative but it’s the best overall measure of our success.

DeepNet Design

So the secret sauce to this whole endeavor is in the DeepNet network design. Since we’re working on a constrained system, Artik 10 (Cortex A-series quad-core), running image recognition in real-time, we had to constrain the DeepNet’s size, both in depth and network width. What we’ve found with this process is that there’s an element of alchemy and heuristics both in network design as well as dataset creation. But in the end you can prune an aweful lot of the network and still get great results in accuracy and hit the performance requirements needed to run in realtime on an embedded system.

We played with how many convolutional kernels (width of the network), less kernels means less “features” it can discover, but we’ve found that unintuitively, you can actually get better results. The number of layers or the depth of the CNN is also drastically reduced to what you’d find in a larger server based or unconstrained solutions. The size of the convolution kernel is also surprisingly small as well.

On top of all of this, we have an Artificial Neural Net (ANN) for the final classification. For the activation functions, we use Rectification for the CNN and for the ANN fully connected layers, we will often play around with various activation functions, this is highly dependent on the application itself.

The Result

The result? We have a highly accurate real-time image recognition system that works offline on highly constrained devices. If you want to add hand signals to a camera to control camera functionality, do you want to add event triggers to low-cost toys, you name it!

 

 

Yup, the Hololens finally arrived and it was worth the wait. It doesn't have a ton of content yet, of the platforms we have so far --HTC Vive, Oculus, and Hololens--the Oculus has the most mature content and the Vive seems to have the most by virtue of Steam.

Played around with the Hololens setting up holograms around our conference room, fun. The first thing we noticed is that the field of view of the projected items gets a little cut-off, but the overall AR is impressive.

crossmenu