The MIT CSAIL team used a simulated, anthropomorphic hand with 24 degrees of freedom and showed evidence that the system could be transferred to a real robotic system in the future.

The system uses a model-free reinforcement learning algorithm with deep learning and something called a “teacher-student” training method. The “teacher” network is trained on information about the object and robot that’s easily available in simulation but not in the real world. The knowledge of the “teacher” is then distilled into observations that can be acquired in the real world, such as depth images captured by cameras, object pose, and the robot’s joint positions.

The robot could reorient a large number of objects it had never seen before and with no knowledge of shape. It can manipulate small, circular-shaped objects, such as apples, tennis balls and marbles, with a nearly 100 percent success rate. However, with more complex objects, like a spoon, a screwdriver or scissors, its success rate falls closer to 30 percent. Since success rates varied with object shape, in the future, the team notes that training the model based on object shapes could improve performance.