“While grasping and picking up an object is a basic task for humans, it has proved incredibly difficult for machines,” says Jurgen Leitner, Ph.D., a professor of electrical engineering and computer science at QUT. “The world is not predictable—things change and move and get mixed up and, often, that happens without warning. So, robots need to be able to adapt and work in very unstructured environments if we want them to be effective.

“We have been able to program robots, in very controlled environments, to pick up very specific items,” explains Leitner. “However, one of the key shortcomings of current robotic grasping systems is the inability to quickly adapt to change, such as when an object gets moved.”

The neural network approach works by predicting the quality and pose of a two-fingered grasp at every pixel. By mapping what is in front of it using a depth image in a single pass, the robot doesn’t need to sample many different possible grasps before making a decision, avoiding long computing times.

“In our real-world tests, we achieved an 83 percent grasp success rate on a set of previously unseen objects with adversarial geometry, and 88 percent on a set of household objects that were moved during the grasp attempt,” claims Leitner. “We also achieved 81 percent accuracy when grasping in dynamic clutter.”

According to Leitner, he and his colleagues overcame a number of limitations of current deep-learning grasping techniques.

“For example, in the Amazon Picking Challenge, which our team won in 2017, our robot CartMan would look into a bin of objects, make a decision on where the best place was to grasp an object and then blindly go in to try to pick it up,” says Leitner.

“Using this new method, we can process images of the objects that a robot views within about 20 milliseconds, which allows the robot to update its decision on where to grasp an object and then do so with much greater purpose,” claims Leitner. “This is particularly important in cluttered spaces.

“This research enables us to use robotic systems not just in structured settings where the whole factory is built based on robotic capabilities,” Leitner points out. “It also allows us to grasp objects in unstructured environments, where things are not perfectly planned and ordered, and robots are required to adapt to change.”

Leitner says potential applications for this technology include assembly lines, warehouses and fruit picking. “It could also be applied in the home, as more intelligent robots are developed to not just vacuum or mop a floor, but also to pick items up and put them away,” he explains.