General Electric Co. has started using augmented reality devices as the company takes a major plunge into the use of artificial intelligence and virtual reality.

At the 2016 GE Minds + Machines conference held this week in San Francisco, Colin Parris, the vice president of GE Software Research, demonstrated how employees are talking to machines and interacting with them using Microsoft’s HoloLens augmented reality device.

GE has created so-called “digital twins” of the machines that it sells — a steam turbine for instance — that are digital replicas of actual machines at customer sites. The company has created a software system that allows customers to speak to the digital twin and ask it questions about potential parts breakdowns, financial forecasts and the best way to fix problems.

The digital twins are loaded with data they can crunch to provide the best advice — which is given in real language not unlike Siri on the iPhone.

 

“This is happening now,” Parris, who works in Niskayuna, said after he talked to a digital twin of a steam turbine at a customer site in Southern California. “What you saw was an example of the human mind working with the mind of a machine.”

The digital twin can run thousands of simulations at a time using environmental and operational data to predict breakdowns or other events.

And when a machine needs to be fixed, GE and its customers can use augmented reality to look inside those machines without having to actually touch them.

Parris put on a Microsoft HoloLens — an augmented reality headset — to superimpose the digital twin over a picture of the actual steam turbine. The HoloLens allowed him to open up the turbine and look at the parts — and see exactly which part may need replacing.

Parris said GE has been partnering with Microsoft on augmented reality technology. He says that AR as it is also called can help GE executives redesign a factory floor by moving parts around in augmented reality.

It can also help with training and production, helping to teach workers how to assemble parts even before they ever step on a factory floor.