A Vision for Augmented Reality & Augmented Intelligence

As you may know from my previous posts, my primary motivation is improving shop floor performance and consistency – usually with technology. My goal is to deliver functionality that allows our operations team to better visualize and manage manufacturing performance with real-time metrics. This has resulted in some interesting applications now being deployed globally across our organization.

Historically, though, I’ve tended to stay away from projects involving the kind of terminology that might put those outside of IT and OT to sleep. I typically pursue quick, simple solutions that can be swiftly deployed and then expanded. Call it skunkworks or rapid experimentation, the point is that I’ve focused on the most expedient ways to improve operational performance, not necessarily on adopting bleeding edge technology.

Yet here I am writing about augmented reality and augmented intelligence.

Why? These rapidly evolving technologies offer the greatest opportunity to improve operator productivity — getting quality products out in a predictable, timely fashion with minimal waste and rework — at a time when that is one of the organization’s biggest priorities.

The Manufacturing Skills Crunch

Employee retention is a challenge for all industries, but in manufacturing we feel the pain of mobility and attrition acutely. Finding someone who knows how to use Excel or code in Python is one thing. Those skills are transferrable among many white collar jobs and available in relative abundance. Trying to find someone who understands the ins and outs of assembling a modern automotive component is quite another.

In our automobile interior manufacturing facilities, for example, our operators often work with leather. Managing that material is quite different than handling basic plastic. These individuals work in a complex environment of specialized processes such as component wrapping, variable temperature molding, and 3D robotic sewing and stitching. Training for these roles can take many months — or even years — to master.

Losing these skilled workers has been a particular challenge over the past four years. Between the pandemic and ongoing supply chain issues, numerous manufacturing companies have had to halt operations and temporarily lay off skilled operators for a time. During these furloughs, employees may leave for another job.

Looking Beyond Metrics and Automation

The question for my team is: how can we apply technology to improve this situation? The production monitoring and real-time metrics we’ve enabled give our production supervisors more data with which to manage the production environment than they’ve ever had before. We also recently introduced electronic tracking of training and certification , confirming that as employees “badge into” their work center, they have the requisite skills for the environment. (This replaced an old paper system that was difficult to maintain, at best.)

But to address our labor challenge, we must go well beyond improved visibility or increased digitization to equip our line operators to function at a higher level. That’s where augmented reality and augmented intelligence start to make sense.

In some facilities, we’ve already begun displaying work instructions for parts as they are presented on the line. When a new part is scheduled, new instructions appear on a nearby monitor often powered by a Raspberry Pi. We’ve found that this can serve as a refresher to the operator in case it’s been a while since they’ve worked with a specific part or a particular configuration hasn’t been run recently. As a next step, we’re adding short videos as a way to provide training on the production line.

The Augmented Opportunity

With augmented functionality, we can take this much further.

Deploying augmented reality, we can layer drawings, documentation, and other relevant information on top of a physical component or environment. This significant upgrade over simply displaying static instructions on a screen adjacent to the line can deliver improvements in productivity, accuracy, and safety.

Augmented intelligence will be the next step. Defined by Gartner as “a human-centered partnership model of people and artificial intelligence (AI) working together to enhance cognitive performance,” augmented intelligence will give us the opportunity to blend the unique capabilities of human and machine learning to produce high-quality parts more quickly and effectively. An operator can ask this system questions about a process rather than having to track down an expert or consult a manual, for example, enabling just-in-time learning and discovery.

I see multiple use cases for augmented reality and augmented intelligence in our production environments. I’m championing their development for:

• New employee training. Augmented reality capabilities will enable the gamification of training. As more people experience consumer VR products, they are gaining some understanding of the basic approach. The immersive experience AR offers will be much more engaging, interactive, and effective than reading a manual, watching a video, or even shadowing someone.
• Remote problem solving. Control engineers from our technical centers design, develop, and implement the equipment used in our manufacturing facilities. Deploying and fine-tuning these intricate machines takes time, requiring these experts to travel for extended periods of time and put their other projects on hold. Using AR in combination with network connectivity, these engineers can “look” through the lens of a local engineer or technician without leaving their offices, providing their expert insight while eliminating travel costs and increasing productivity.
• Complex part and component assembly: As I mentioned, our operators are performing highly complex tasks to produce components and systems for the automotive industry. Integrating a “light curtain” concept (to highlight the correct bin from which to pull a part, for example) and other error-proofing techniques with AR headsets enable simpler and more efficient component selection during the assembly process.
• Real-time inspections: Historically, product inspections have been performed primarily at the end of the line. While the most expedient approach (with available technology at the time), it did not allow for the detection of work-in-progress problems. In addition, manual testing and data entry onto a whiteboard or into a spreadsheet can result in errors. Augmented reality and intelligence offers a way to more inspect production in real-time. Since the operator is basically connected to a computer through the ER goggles, activities can be verified much easier as the work is done. Identifying and correcting visible defects is critical for “Class A” automotive surfaces or parts that the customer sees and touches.

The Road Ahead

As an automotive industry CIO, I have a bias toward action, seeking out solutions that are ready to deploy and make an impact. But it’s clear to me that the possibilities of augmented reality and augmented intelligence are quickly becoming reality. As we look ahead, the business case for designing and implementing a more efficient and employee-friendly manufacturing environment, with the aid of these technologies, is coming into focus.

And that’s just a start. Given our focus on engineering and design excellence and the need to compete in the global marketplace, we’re exploring a variety of new technologies for production and sales. Like many CIOs, I’m evaluating generative AI options. Our skunkworks project in this area is a Proof of Concept to implement a ChatGPT interface within our more protected cloud environment. Generative AI will only add value if we can use company data for analysis, and we will not allow our data to be absorbed into the large language models (LLMs) of public generative AI tools.

The next year promises to open many new doors for technologists as we learn to incorporate consumer-like experiences (whether that’s gamification or gen AI tools) into our systems. Buckle up!