Traditionally, it’s not unusual to see people wear goggles and other types of safety glasses in factories. But, a new class of wearable devices is becoming increasingly popular on assembly lines.
Next-generation eyewear embedded with mixed reality technology is enabling people to leverage the power of extra sensory perception on the plant floor. It allows them to work more efficiently and more productively than ever.
One of the most common applications is augmented reality (AR), a technology whereby objects in the real-world are “augmented” by computer-generated perceptual information, including visual, auditory and haptic sensations. The primary value of AR is that it brings components of the digital world into a person’s perception of the real world.
However, AR can do more than just display data. It can include immersive sensations that are perceived as natural parts of an environment. Manufacturers can use AR to design assembly lines, improve plant floor layout, train new employees, inspect products for defects, streamline material handling and deliver paperless work instructions anywhere on the shop floor.
“AR can superimpose work instructions or safety tips directly into the worker’s field of view,” says Jerry Foster, chief technology officer at Plex Systems Inc., a leading supplier of manufacturing execution systems software. “It can guide them through complex processes.
“AR can show employees critical stats and metrics on a machine, workstation or device just by looking at it,” notes Foster. “And, it can provide a space for testing without damaging physical property or endangering employees.”
Augmented reality applications started a few years ago at a handful of large manufacturers. As the technology has matured and become less expensive, more companies are now using AR in smart factories.
“Augmented reality technology has two main benefits for manufacturers today,” says Jason Andersen, vice president of business line management at Stratus Technologies. “First is operator training. “AR software can reduce training time, increase safety and make it easier to transfer institutional knowledge.
“AR can also assist operators [with complex assembly tasks],” claims Andersen. “If you are executing a process and there is some benefit to getting data about the steps already completed on a product coming down the line, there should be a productivity increase.”
“Several large manufacturers are already at the forefront of applying AR technology in their factories,” adds Charles Shearrow, Ph.D., associate professor of advanced manufacturing at the University of Harrisburg. “Lockheed Martin is using it for training; Caterpillar and Mitsubishi Electric are using AR for maintenance; and Deere is using the technology for assembly applications.”
“Augmented reality is the convergence of real world and computer-generated environments,” says Mike Moran, director of engineering at Morey Corp., an electronics manufacturing services provider. “These environments include everything you would normally experience at a particular place and time, along with overlays that are comprised of visual, audible and touch elements like haptic feedback.
“Typically, an AR user will wear a pair of glasses or goggles that provide see-through visual capability mixed with graphic overlays that appear to be suspended in the surrounding real-world environment,” explains Moran. “Often, these devices will complement the enhanced visual environment by providing synchronized audio cues using ear buds to further enhance sensory augmentation.
“Many people hear the term augmented reality and think of virtual reality or immersive video games,” claims Moran. “This could not be further from the truth in terms of what AR really is and can do.”
“Augmented reality is the overarching term that describes a user experience that enhances the user’s real world with contextually relevant digital information,” adds Jay Kim, chief strategy officer at Upskill, a company that provides AR software for companies such as Boeing, General Electric, Lockheed Martin and Schneider Electric. “Within augmented reality, two common spectrums of experiences exist for industrial environments: assisted reality and mixed reality.
“Assisted reality provides contextually relevant digital information in a nonimmersive, 2D format,” notes Kim. “Images are not registered to objects, and the visual does not change based on the user’s position or field of view.
“This experience is similar to a heads-up display found in car windshields or fighter jets, delivering glanceable information to assist the user in performing the task at hand,” Kim points out. “Common sources of information include text, images, engineering drawings, symbols and videos.”
Mixed reality is a blend of physical and digital worlds in a wearable headset, resulting in an immersive and 3D user experience with full understanding of the environment.
“With mixed reality, there is no limit as to whether the experience is 2D, 3D or some mix of both,” says Kim. “The digital information can be connected to objects in the real world and can persist or follow the user’s movement.
“Common sources of information delivered to this user experience include all that assisted reality can offer, plus 3D models, engineering drawings and schematics in its native form, high-fidelity videos and PDF-based work instructions,” explains Kim.
Contrary to popular belief, augmented reality is not the same as virtual reality (VR), which is popular with video game users.
“While VR if fully immersive, it leaves out the real-world environment that is critical to [assembly line] operations,” says Moran. “Consider the
ability to see and hear information and instructions related to whatever you look at in a particular environment. This is only possible in an AR realm and can provide disruptive capability to improve efficiency in real-world jobs and businesses.”
“In virtual reality, the user is being immersed in a totally different reality, which is different from the physical world,” adds Liliya Pautova, director of business development at Proceedix Inc., a company that specializes in work instruction software. “The use cases are therefore entirely different. AR and assisted reality can be deployed when people are doing things, such as assembling products. VR can only be deployed in artificial settings for training and designing.”
AR’s Role in Smart Factories
According to many experts, augmented reality is one of the most important elements of a smart factory, where assets are becoming increasingly intelligent and connected across disciplines. Manufacturers operating smart factories can use AR to guide employees through the production cycle and validate that assembly tasks were executed correctly.
“Smart factories are the ultimate environment for AR applications,” says Moran. “When all of the factory is equipped with sensors and automation, the amount of data and analytics that can be made available to workers is nearly endless.
“Being able to tie together environmental conditions, inventory levels, process state, assembly error data, utilization and throughput metrics in a context-dependent manner (where you look or walk) creates an unprecedented awareness of the factory performance at any given point in time and location,” claims Moran.
“AR is an important component of smart factories,” adds Dan Cefaratti, North America Dynamics 365 field service lead at Avanade, a joint venture between Accenture and Microsoft. “Ensuring that machines are at full capacity means there can be little to no downtime, without any disruption to the production process or issues related to it.
“AR-enabled devices are always connected, which means they can pick up data from Internet of Things (IoT) sensors in a factory setting to provide input and analytics to enable real-time decision making,” explains Cefaratti. “These devices can also be used to scan various markers on machinery to bring up manuals, diagnostics or integrate with security protocols.”
“AR can be applied in a number of different areas in smart factories, such as product design and development, assembly lines, logistics, warehousing, equipment maintenance and support,” says Ajay Chidrawar, vice president of global product management and customer success at CGS, a software company that specializes in augmented reality applications.
“With IoT sensors and scanners, a massive amount of data is being generated every minute,” notes Chidrawar. “This data can then be uploaded to the cloud, crunched in real time, and delivered directly to workers and managers anywhere in the factory to provide insights and help them make better decisions.
“AR can leverage this type of information and overlay it in the context of what the worker is doing, whether it be on an assembly line or in a warehouse,” Chidrawar points out. With augmented reality technology, employees can stay attuned to their physical context without fumbling with handheld papers or devices. They don’t need to carry a heavy manual across the plant floor or glance back and forth between a tablet screen and machinery. They can receive hands-free contextual information at a glance.
“AR in a smart factory setting offers the opportunity to deliver information to a worker at just the right moment, so they don’t have to interrupt the flow of their work to refer to devices,” says Hilary Dixon, a user experience designer at Nerdery, a digital consultancy that works with manufacturers. “It adds efficiency and reduces friction.
“Augmented reality can also facilitate clearer communication between groups of people in a physical space,” claims Dixon. “For instance, an engineer could be wearing a HoloLens and a co-worker could view the work through a tablet to help them troubleshoot a problem.”
“AR plays a critical role in smart factories by enabling [employees] to work in conjunction with the additional intelligence of technologies like artificial intelligence, data analytics and digital twin,” adds Upskill’s Kim. “AR can be seen as an extension of smart factory technologies to the workforce, enabling people to interact with the smart infrastructure around them.
“In smart factories, IoT-enabled sensors and machines report real-time status of everything ranging from manufacturing equipment to products, and provide insights on what needs to be done when,” says Kim. “AR empowers people to ensure that the work is performed right every time, with no time lost browsing cumbersome paper-based processes and instructions.”
Basic Elements
Augmented reality technology requires several types of hardware and software components. Hardware typically consists of wearable devices, such as smart glasses or headsets. However, it can also include smartphones and tablet computers.
“For assembly line applications, AR software typically consists of the component that is running on the devices themselves, which delivers the user experience best suited for the task, and the services layer component that is connecting the user experience with the content that is powering it,” says Kim.
“In an assembly environment, this is typically a set of work instructions, engineering diagrams or checklists to assist the hands-on worker,” Kim points out. “This software can be developed using a platform or built from scratch.”
“The software required depends on how you plan to solve your use cases,” adds Plex Systems’ Foster. “Just like in regular web and software applications, you can purchase the solution or build your own.
“If you purchase your solution, the software would be delivered by the provider,” says Foster. “However, if you plan to do any in-house development, you will need to use a dedicated augmented reality toolkit, such as the popular Unity platform.”
“Hardware and software aside, there are typically basic infrastructure requirements to create the best AR experience,” adds Kim. “Good Wi-Fi connectivity is great to have, although not required. But, there would need to be somewhere the devices can establish connectivity to production execution systems.
“Device security is often important,” warns Kim. “Mobile device management systems, such as AirWatch, are commonly deployed to secure the devices and the applications running on AR hardware.”
Manufacturing engineers can choose several types of hardware for AR applications, including full headsets, such as Microsoft’s HoloLens, and smart glasses, such as Google’s Glass.
HoloLens has been available for several years and Microsoft recently unveiled a new version called HoloLens 2. The lightweight device features a self-contained computer and Wi-Fi connectivity.
Ford Motor Co. was one of the first manufacturers to use HoloLens at its manufacturing R&D lab near Detroit. The automaker foresees even more applications for HoloLens 2.
“The second iteration is exciting, because it more than doubles the field of view to 52 degrees horizontally,” says Marty Smets, a technical expert in human systems and virtual manufacturing at Ford. “The battery life and accuracy of the spatial tracking feature have also been improved on HoloLens 2. It simultaneously locates and maps the environment.”
Earlier this year, PTC Inc. released new AR software that supports HoloLens 2.
“Vuforia solutions give industrial customers an efficient and effective way to create interactions that capitalize on HoloLens 2’s new gestures, voice enhancements and tracking capabilities without the need for extensive programming or costly custom designers,” claims Mike Campbell, executive vice president of PTC.
“One of the key value drivers for AR in the industrial space is improving worker efficiency with hands-free procedural guidance and instruction,” explains Campbell. “Vuforia Studio’s efficient AR authoring environment and HoloLens 2 usability and comfort enhancements are a winning combination for manufacturers.”
Smart glasses are another option that is especially popular for informed reality applications. They provide a sensation that is like wearing a little screen built into your glasses that goes wherever you go.
One of the first devices on the market was developed by X Development LLC, a subsidiary of Alphabet Inc. (Google). Glass Enterprise Edition features a heads-up display that’s easy to use. The smart glasses have been adopted for assembly line applications by manufacturers such as AGCO, GE and Volkswagen.
AGCO’s state-of-the-art factory in Jackson, MN, which assembles a wide variety of Challenger and Massey Ferguson farm tractors, has been using Glass for three years. Assemblers at the 2017 Assembly Plant of the Year wear the lightweight device to scan a machine’s serial number to instantly bring up a manual, photo or video they may need to build high-mix products. In addition, employees can use voice commands to take notes and leave them for the next shift worker, allowing for a more seamless transfer and increased productivity.
Assemblers begin each task by saying, “OK, Glass, proceed.” Or, they can swipe along the right side of the frame and proceed to the next image. Assembly steps or quality checklists appear in the upper right corner of the tiny screen.
Glass tells them things such as what kind of bolt is needed, which fastening tool to use and how much torque is required. If they encounter a problem, such as a damaged part, employees can take a picture of it and ask for help.
“Smart glasses hold two unique advantages over mobile technology in an industrial context,” claims Proceedix’s Pautova. “They are the best devices when you need to work hands-free. The embedded cameras also perfectly capture the wearer’s perspective when establishing a video streaming session for remote assistance.
“Heads-up-display smart glasses are easy to use and have matured as a technology,” explains Pautova. “They can be controlled with gestures or with voice commands, enabling true hands-free working. At AGCO, our work instruction software increases the efficiency, quality and safety of its assembly processes.”
Several alternatives to Glass are also available, including Blade and M300XL from Vuzix Corp., which pioneered AR technology in the late 1990s when it developed products for the U.S. military. Today, the company supplies manufacturers such as Airbus and Deere with lightweight products that are designed to be worn on assembly lines all day.
Blade resembles a pair of sunglasses, but incorporates patented wave-guide technology. It features an integrated smart display that offers the user a hologram-like visual experience. The image is projected onto the lens of the glasses and is transparent so that the wearer’s field of vision is not restricted by the display. Assemblers can simultaneously perceive what is happening in their environment.
“Blade provides a natural view of the real world while concurrently allowing AR content to be simply viewed and interactively managed,” claims Paul Travers, president and CEO of Vuzix. “It features transparent optics designed to overlay visual information.”
Vuzix’s newest product is the M300XL. Its frames can be adjusted for left or right eye use; mounted onto a hardhat, headband or safety eyewear; and worn with prescription lenses or no lenses at all. The smart glasses feature an Android operating system and a 16-megapixel camera that enables users to quickly scan bar codes.
A relatively new player in the AR hardware market is RealWear Inc. It offers a rugged, hands-free Android device called the HMT-1 that is popular for in-the-field applications in the utilities and oil and gas industries.
The HMT-1 weighs 350 grams and enables people to follow work instructions, watch videos or communicate with experts even in extreme noise, temperature and darkness through voice navigation. An adjustable boom houses a monocular display, which appears as a 7-inch display relative to the eye.
“The device can be used to play video, look at work instructions or augment the worker’s environment in real-time,” says Andy Lowery, CEO of RealWear. “Intended for remote video collaboration, technical documentation, industrial IoT, data visualization, assembly and maintenance instructions, the HMT-1 is a safer, faster and smarter way for workers to get their jobs done in harsh and loud environments.
“The HMT-1 offers more utility per device than HoloLens,” claims Lowery, a former Raytheon engineer who founded RealWear several years ago. “Since it was launched last year, the HMT-1 has been well received in the marketplace. We try to focus on the user experience and a hands-free interface, so there’s no need for people to gesture, press buttons or swipe.”
The HMT-1 and other popular AR devices, such as Glass, HoloLens and M300, are compatible with software developed by Ubimax Inc. Its Frontline platform is designed around four products: xPick (pick by vision), xMake (guided assembly), xInspect (guided inspection) and xAssist (remote support).
“These foundation products have been seamlessly integrated into a full, end-to-end, Frontline solution platform that addresses the deployment, technology integration and operational management inefficiencies that have previously plagued the wearable technology marketplace,” says Percy Stocker, president of Ubimax Americas.
“Similar to the technological leap that Microsoft introduced with Office, Frontline is a unifying platform for [manufacturers] seeking to gain a competitive edge by deploying wearable computing technology to their frontline workers,” claims Stocker.
Other companies that produce AR hardware include DAQRI (Smart Glasses and Worksense), Magic Leap (Magic Leap One), Sony (SmartEyeglass) and Toshiba (dynaEdge).
Bottomline Benefits
Manufacturers that have implemented augmented reality claim the technology has numerous competitive advantages. Three main benefits of AR include the following:
- It enables workers to remain hands-free with a constant flow of information so they can complete their tasks more efficiently, accurately and safely.
- It connects the smart factory to the workforce and captures insights from their point of view for smarter operations and production planning.
- It drives increased throughput, first-time quality rates, workforce utilization and compliance.
“Augmented reality can play an important role in several areas of manufacturing,” says Morey’s Moran. “One key application is the training of factory workers in a simulated environment using virtual equipment. Seeing and interacting with this virtual environment is similar to hands-on training without needing to allocate real, revenue-generating machinery to training activities.
“Another key area is in assisted manipulation and analysis of items moving through a line,” Moran points out. “For example, a worker could be instructed of low supply levels on a component reel with a level indicator when they look at a particular machine. Likewise, the employee may be informed of an incorrectly oriented component during a visual inspection.
“By giving people the ability to see what is happening upstream and downstream from them, assembly operations can be greatly enhanced,” claims Moran. “In effect, this awareness gives workers a linked together sensation without needing to leave their station, which in turn, leads to greater productivity.”
“Augmented reality provides manufacturers with a clearer look into their business’ processes,” adds Jeff Ralyea, president of the manufacturing division at ECi Software Solutions. “It allows manufacturers to complete their processes much faster and more efficiently than previous methods.”
Traditionally, manufacturing operations have gathered data manually, causing delays and miscommunications. Now, with the ubiquity of data capture solutions and the emergence of AR, complex processes, especially those surrounding assembly, are being made simpler through the accessibility of accurate data and real-time visibility.
“Frontline workers are more effective and need less training as they have the ability to obtain knowledge and leverage it when and where it’s needed,” claims Mark Wheeler, director of supply chain solutions at Zebra Technologies Corp., a company that specializes in data capture hardware ranging from bar code readers to wearable computers.
“Workers can now be held accountable for their tasks and decision-makers are able to make adjustments, due to increasing insight into how a part was maintained, delivered and assembled,” says Wheeler. “In this sense, data and AR will allow manufacturers to become more intelligent as time goes on.
“Augmented reality technology can help manufacturers improve productivity and streamline assembly operations by enabling them to access and act upon data in real-time,” explains Wheeler. “For assembly operations in particular, AR can outline the correct sequence of events for the assembler to reference, minimizing the risk of error and enhancing overall efficiency.”
AR in Action
Augmented reality is transforming the way that some industrial organizations operate today. It’s connecting workers with critical information to do their jobs in real-time, while improving productivity, quality and efficiency.
“Assembly operations are streamlined simply by eliminating paper-based instructions and connecting the workforce to digital sources of record, resulting in significantly easier change management and enforcing standard work,” says Upskill’s Kim. “Companies within the aerospace, automotive, life sciences and telecommunications industries have already realized the benefits of AR and are starting to deploy [the technology] at scale.”
One manufacturer leading the AR charge is AGCO. Its factory in Jackson, MN, which was an early adopter of Glass wearable technology, has achieved impressive results.
The ability to have access to instructions in quality, assembly and paint prep has led to reduced processing times, reduced defects and reduced training costs. AGCO has seen over a 30 percent reduction in inspection times, a 25 percent reduction in production time and the ability to train staff 300 percent faster.
“Glass was introduced as a solution to make employees’ jobs easier and safer, while driving higher quality to our product and our processes,” explains Peggy Gulick, director of business process improvement. “In the end, we have accomplished both.
“We found the greatest value from using Glass has been in the assembly and quality areas, through the easy and quick hands-free access to the instructions and checklists necessary to assemble our products.
“In addition, we have discovered that training with smart glasses is a grand slam,” claims Gulick. “New product launches and new hire training are easily administered and audited for success.
“Staff is trained faster on cross-functional operations using Glass, reducing the average learning curve time from 10 days to three,” Gulick points out. “Knowing that smart glasses are a lean tool, and not an industry requirement or cool factor, we have achieved a 30 percent reduction in processing times and a 50 percent reduction in new hire and cross-functional employee training time.”
Boeing is another large manufacturer that’s bullish on AR technology. It has deployed smart glasses and Upskill’s Skylight platform for wiring harness applications. With AR, Boeing has been able to reduce error rates and cut production time by 25 percent.
“Installing electrical wiring on an aircraft is a complex task that leaves zero room for error,” says Paul Davies, a research and technology associate technical fellow at Boeing. “That’s why [we implemented] augmented reality as a solution to give technicians real-time, hands-free, interactive 3D wiring diagrams right before their eyes.”
“A person working in an industrial setting has a lot of distractions in that environment and a lot of data to think about and process,” notes Davies. “Traditionally, technicians had to look at and interpret a two-dimensional 20-foot-long drawing and construct that image in their mind and attempt to wire based on this mental model.
“By using augmented reality technology, [assemblers] can easily see where the electrical wiring goes in the aircraft fuselage,” Davies points out. “They can roam around the airplane and see the wiring renderings in full depth within their surroundings and access instructions hands-free.
“Our theory studies have shown a 90 percent improvement in first-time quality when compared to using 2D information on the airplane, along with a 30 percent reduction in time spent doing a job,” claims Davies.
