A person can grab a random part out of a bin full of similar or varying items without even thinking about it. For a robot, however, such a task is much more difficult.

For decades, robots in controlled environments like assembly lines have been able to pick up the same object over and over again. More recently, breakthroughs in computer vision and motion control software have enabled robots to make basic distinctions between objects.

But, robots still struggle to understand the shape of objects. If a part is clear, flexible, reflective, shiny or metallic, robots simply can’t compete with humans. Components that are stacked, interlocked or tangled also present problems.

Part Orientation

In structured bin picking, part orientation can typically be predicted so that approximate data can be extrapolated before a pick is even calculated. In random bin picking applications, however, parts can be sitting in a number of orientations and can be stacked in multiple configurations.

“Seemingly simple tasks like bin picking are still done primarily by human workers today,” says Jared Glover, Ph.D., CEO of CapSen Robotics, a startup company that recently developed a 3D vision system and motion planning software for robotic bin picking applications. “That is because the parts are presented randomly in a pile, [which] makes the computer vision job of finding the part locations challenging.

“[That] also means the robot must plan and execute a different picking motion each time it reaches into a bin, deciding how to grasp the object while avoiding collisions with other objects along the way,” explains Glover. “On an assembly line, the part being picked must often then be fed into another machine or assembly in a precise way, requiring additional real-time geometric planning and execution.

“Random bin picking is a challenge for two main reasons,” Glover points out. “First, it’s difficult to have a general computer vision algorithm that can tell you where objects are, no matter what they’re shaped like. The other challenge is figuring out how to pick parts presented in different ways and avoid colliding with things. Deformable objects in particular are still challenging for robots.”

“Even with new robot and camera technologies, vision system challenges persist,” adds Ryan Guthrie, executive vice president at TM Robotics (Americas) Inc. “The robot arm must provide full six-axis movement in order to use varying approach vectors to reach into a bin without hitting the box sides, and then pick up items that can be lying on top of each other in a wide range of positions and orientations.

“The camera must be able to scan, process and communicate data quick enough to coordinate the robot’s actions,” warns Guthrie. “And, the images captured must be clear enough to show more than just outlines of the items in the bin. For the robot gripper to effectively approach a targeted item, the position and orientation of items that may be jumbled or overlapping must also be identified.”

According to Guthrie, what makes random bin picking so difficult is that engineers must find a way for the robot to put together a complete picture of more than just what is in a bin or tote. Engineers must consider things such as: Are there any obstructions? How is the tool going to pick the part? Will the edge of the box or bin get in the way of the tool? Will the knuckle (the fifth joint on a six-axis robot) collide with the box geometry once a part is detected?

“All of these things need to be calculated, and calculated in milliseconds in order for a project to be viable,” Guthrie points out. “You are no longer just identifying the part. You also need to identify the box and the tooling, and have an understanding on what physical space the robot itself occupies.”

Traditionally, bin picking has been a huge systems integration challenge that requires multiple technologies to work together. In addition to robots and finger or suction grippers, engineers need the following:

• A means to create a 3D model of a part (scans or CAD conversion).
• A 3D sensor to scan the environment.
• Image analysis software to locate the part and detect obstacles or obstructions, such as adjacent parts or foreign objects.
• Motion control software to enable the robot and end effector to pick and place the part without colliding with environmental obstacles.

“All of this needs to work in an industrial environment that has many changes in lighting and part variations,” says Eric Truebenbach, director of business development for UR products at Energid Technologies Corp. “This is a complex systems integration problem that few companies are capable of solving.

“However, there is no general algorithm that a user or integrator can apply to guarantee a collision-free path from an infinity of possible part locations to the place location,” notes Truebenbach. “As a result, programmers [often] spend weeks trying to create all of the permutations. But, in the end, exceptions still cause mispicks and collisions.

“The only way to make bin picking reliable is with real-time path planning software that can take into account the part pose, surrounding parts, the end effector, robot, bin, place target and environment,” adds Truebenbach.

New Tools and Technology

Today’s bin picking technology is dramatically different than what was available a few years ago. Prices have decreased, software has been simplified and processing speeds have increased.

“Bin-picking technology has progressed significantly as the accuracy, resolution and performance of sensors have improved,” says Jason Niu, marketing manager at Bluewrist Inc., a Canadian company focusing on intelligent automation systems using 3D vision cameras. “This greatly aids in better part identification that is critical to calculate the optimum pick coordinates.

“Increases in computing power, such as multicore CPUs and GPUs, enable faster processing of 3D point clouds, which makes 3D vision bin picking feasible in production environments where short cycle times are mandatory,” Niu points out. “Today’s bin-picking systems are easier to set up and configure, and they’re more flexible to adapt to changing requirements. For example, [when there are part or component changes] systems can be reconfigured on the software level.”

“What would have taken the processing power of a super computer 10 years ago can now be processed in mere milliseconds with today’s commercially available PCs,” claims TM Robotics' Guthrie. “This brings the price point to within the standard integrator’s budget. The extra processing power also allows for more intelligent tools to make the user interface easier for a nonvision expert to understand.

“More and more manufacturers are now custom-configuring their systems to match specific models of robot arms,” says Guthrie. “That helps establish a relationship between the physical space and the virtual space. User interfaces are also becoming easier to set up.

“The curve is very similar to machine vision,” notes Guthrie. “Twenty years ago, you needed a doctorate degree to not only understand, but to program a vision system. Now, we have integrators that are new not only to vision, but to robotics in general, tackling bin picking applications successfully.

“The next major step in bin picking is eliminating the need to preteach the parts,” says Guthrie. “The holy grail is robots and vision systems having the intelligence to pick any part out of any box without any training.”

While that’s still a few years away, engineers can already choose a variety of new products that provide the high speed, accuracy and consistency needed for robotic bin-picking applications.

For instance, TM Robotics and Toshiba Machine recently launched the TSVision3D system. It incorporates two integrated, high-speed stereo cameras that are capable of 30 frames per second for continuous, real-time 3D images. The cameras manage image capture, processing and parallax operations to identify items’ positions, and the vision software adds easy model registration without requiring complex CAD data.

To enhance the camera’s accuracy even further, a projector shines a random light pattern into the parts bin, which highlights the surfaces of the items inside and gives the camera additional position and orientation data for more accurate identification.

“[In the past], three-dimensional vision technologies required extensive expertise and were difficult to use,” says Guthrie. “[Our system] was developed so that anyone, even with minimal training, can understand and implement it.

“Cycle time for items to be picked can vary depending on the situation, with typical cycle times of 3 seconds, providing an optimized balance between processing speed and accuracy,” claims Guthrie. “If only one workpiece is present per image or per tray, cycle time can be as fast as 0.7 second, while an image full of workpieces can still be processed in as little as 5 seconds.”

“This new approach to 3D vision combines the latest vision system technologies and software to overcome stumbling blocks previously experienced with 2D vision systems,” adds Nigel Smith, CEO of TM Robotics. “TSVision3D makes automated bin-picking a reality, even for smaller volume and highly variable applications.

“2D vision sensors have long been the standard for robotic automation systems,” explains Smith. “But, because these sensors can only recognize two-dimensional position and posture, it was often necessary to design and implement a separate preprocess mechanism to remove objects’ overlap and align the parts in order to correctly perform position detection and picking.

“These additional processes worked against the desired process automation and labor reduction,” notes Smith. “The TSVision3D system frees one or two manual operators from a repetitive, boring and fast-paced process that can be performed more accurately and consistently through automation.”

Robot Agnostic Systems

Several startup companies are offering bin picking systems that are robot agnostic. They will work with many types of machines and commercially available 3D cameras.

CapSen Robotics recently developed a sensing and planning unit (SPU) development kit called CapSen SDK. It is both a design tool and a run-time library.

“Its purpose is to allow engineers to quickly design and deploy application SPUs for specific robotic capabilities, particularly for more complex tasks in factories like assembly, packaging and machine tending,” says Glover. “These tasks require not only state-of-the-art 3D computer vision and motion planning algorithms, but also tight coupling between those algorithms.

“CapSen SDK is designed to make such spatial reasoning and execution easy for many such robotic applications,” claims Glover. “The library contains software modules for:

• Scanning—acquiring and measuring 3D object models.
• Detection—finding objects in clutter.
• Planning—robot motions and high-level tasks.

“It is built on top of our highly optimized proprietary software infrastructure with cloud-based data storage, a highly customized deep-learning library, and a GPU-accelerated matrix math module called MultiMatrix,” adds Glover. “Applications built with CapSen SDK have the ability to do more exhaustive searches in order to solve the most challenging geometric optimization problems.”

The company’s first SPU product, CapSen PiC (pick in clutter), automates the physically demanding and repetitive task of bin picking. It enables robots to pick objects from an unorganized bin and place them in a preset pattern. CapSen PiC features integrated vision and motion planning, and is capable of handling objects of any shape.

CapSen Robotics’ technology is still in the pilot phase and is being beta tested at a small spring manufacturer using a six-axis collaborative robot. The pilot application involves assembling hook and spring mechanisms using finger grippers. However, the system will be commercially available by June 2019.

Another new player in the bin picking marketplace is Photoneo. The Slovakian company has developed a system that works with many different robot brands, including ABB, Fanuc, Kawasaki, Kuka, Omron, Stäubli, Universal Robots and Yaskawa.

Photoneo’s 3D vision system can quickly recognize randomly placed parts in big scanning volumes with high resolution and high accuracy. Its new PhoXi 3D camera features a carbon fiber body and can acquire 1,068 by 800 point clouds, plus texture, at up to 60 frames per second with active ambient light rejection.

“PhoXi 3D is the most accurate and fastest 3D camera in the world,” claims Branislav Pulis, vice president of sales and marketing at Photoneo. “It is based on our patented parallel structured light technology and custom CMOS image sensor. The more precise scan you have, the more precise pick you can do.”

The camera works in conjunction with Photoneo’s easy-to-use Bin Picking Studio software.

“The goal is to transform bin picking integration from programming to working with CAD models, using consumer-like controls,” says Pulis. “Users don’t need to have a deep knowledge of robot kinematics, mathematics and programming skills to use the software. If you are able to program robots, you will be able to set up bin picking systems by yourself.”

Photoneo also offers the PhoXi 3D scanner that can scan an entire pallet or box at once. Five models are available and can cover a variety of scanning volumes for applications ranging from small PCBs to big boxes used in automotive production. Because the device is immune to ambient light, it doesn’t need to be shaded.

Another company that has developed a hardware agnostic bin-picking system is Bluewrist. FlexiPick enables a robot to pick up parts randomly placed in a bin and present them to an assembly line.

“Traditional bin picking requires a part CAD model so the software can ‘learn’ the different angle orientation and build a dictionary to generate the best pick coordinates,” says Niu. “During the picking process, the parts in the bin are matched with the dictionary. This can be time consuming. And, the variation between the CAD model and actual part may cause issues. [Our] CAD-free system does not require a dictionary and the system is able to identify the best pick coordinates solely based on the point cloud image."

In addition, Niu says 2D cameras are unreliable in some bin-picking applications, “because their accuracy is affected by changes in ambient lighting and they have problems with detecting low contrast or reflective objects. As the bin becomes empty, 2D vision systems have difficulty maintaining reliability and accuracy that may lead to bin collision. 3D vision systems overcome those limitations.

“Our solution is also hardware agnostic, which means customers are not locked into a proprietary ecosystem,” explains Niu. “There are many different types of 3D vision cameras and robot arms on the market, each with different strengths and weaknesses.

“We can help customers select the best camera for the job,” claims Niu. “In the future, if production requirements change, customers can just switch out the camera or robot arm without the need of a complete revamp.

“The FlexiPick system is powered by our comXtream industrial communications software, because a complete bin-picking solution requires the seamless integration and communication of different hardware,” Niu points out. “comXtream ensures improved compatibility between the different devices in the bin-picking system.”