VIDEO | AI Systems May Be Creating Hidden Risk on the Factory Floor
Manufacturers are moving quickly to adopt AI and automation, but the way those systems are designed may be introducing new risks—especially when it comes to how humans interact with them.
On an upcoming episode of ASSEMBLY Audible, Chris Draper, CEO and co-founder of morriganAI, explains why the placement of humans within AI systems can determine whether those systems are safe or not:
“The challenge in both of those, whether it be vision or any of the physical AI items, is: what is the role of the human in that safety system?
And we think about these in three general forms: in-the-loop, on-the-loop and out-of-the-loop.
Human control in-the-loop is probably the most stated, but mis-implemented, process. It basically says: if my system has something that isn’t proven to be right, it will stop. The human will stop that system and prevent it from going forward unless everything is exactly perfect.
That’s a true human-in-the-loop system. Those systems end up with high levels of safety, ensuring that we don’t get things going through that weren’t right without the human signing off on them.
That, for certain areas, is very good — but it isn’t very efficient. If the human has to sign off on everything, I’m not getting much real automation going.
I then have human out-of-the-loop, which says the system is fully controlled, and the human can’t even step in and fix something. The system’s been designed well enough that it’s running effectively on its own and can track everything correctly.
Looking for quick answers on assembly and manufacturing topics? Try Ask ASM, our new smart AI search tool. Ask ASM
That’s very good — it’s like a Waymo. If you look at the Google Waymo cars, there’s no driver in there, and the person in the back can’t do anything about what’s happening. That’s an out-of-the-loop system.
But the problem is the vast majority of systems we’re implementing right now are on-the-loop systems. What that means is: let the system run until the human catches an error.
Those are incredibly dangerous. They have to be rolled out very thoughtfully.
I argue in most of my strategy work that you should never have a human-on-the-loop system in production. Because a human on the loop in production is like a Tesla—it can theoretically be autopilot.
But if it starts working well enough, the human who is supposed to be your safety feature gets worse. They get comfortable. They start missing errors because they assume it’s right 80% of the time.
And then we just start missing a majority of that 20% we’re supposed to catch.”
Draper expands on this in the full episode, including how manufacturers should rethink AI as a human-centered system and what it takes to deploy it safely. The full episode drops May 7.
Looking for a reprint of this article?
From high-res PDFs to custom plaques, order your copy today!
