Reality check: augmented and virtual technology in the mining industry

Augmented and virtual reality technologies have the potential to improve mine productivity, reduce equipment maintenance costs and protect personnel. So how are Microsoft HoloLens, Occulus Rift and Minecraft being used to inspire the next generation of mining engineers?

virtual reality

Former Boeing researcher Thomas P Caudell may have coined the phrase in 1990, but the concept of augmented reality (AR) was first mentioned by L Frank Baum, author of The Wizard of Oz, in 1901.

In his book The Master Key, Baum conceives of a pair of spectacles that overlay data onto real life, and goes on to joke, with eerie prescience, about the device "being a century ahead of the times".

As Baum predicted, disruptive AR and virtual reality (VR) technologies are now rapidly maturing at both the consumer and commercial level as tech start-ups and global giants such as Microsoft and Google compete for market share in an industry that is projected to be worth $150bn by 2020.

The global mining industry has been an early adopter of AR and VR innovations as it struggles to overcome multiple economic and operational challenges that are as varied as they are daunting.

These include low commodity prices, pressure to produce ever increasing tonnages, a widening skills gap combined with an ageing workforce, the need to protect workers in risky, remote locations, and an increasing reliance on complex, automated equipment that requires safe, efficient maintenance.
AR, VR and remote technologies have the potential to address many of these issues by offering mining operators improved levels of productivity, safety and machinery uptime, improvements in KPIs such as load factor and swing times, and better staff collaboration and knowledge transfer.

Euro vision: EMIMSAR and the Kumba Virtual Reality Centre

AR overlays interactive digital information on top of the physical world, while VR replaces the real world with a simulated one. In a mining context, the former offers major advantages in that it allows training to occur in a real-world environment while also simulating virtual scenarios.

"AR overlays interactive digital information on top of the physical world, while VR replaces the real world with a simulated one."

Using systems developed by the EU-funded project EMIMSAR, miners are able to view augmented reality versions of often complex equipment on handheld computers and helmet-mounted displays.

Sensors on the machinery record and analyse temperatures, rates of acceleration and sample noise from sprockets to enable staff to assess wear on heavy-duty components such as gears and chains.

That data is fed into a knowledge-based maintenance system, where it is combined with background data on components, and machine and component information to create real-time visualisations of the machinery that can be viewed by the miners as they work on the machine.

The EMIMSAR AR system has been deployed by Germany's largest coal mining firm RAG in all of the company's mines for maintenance planning for long wall equipment, belt conveyors and loaders.

Opened in August 2015, the $4m Kumba Virtual Reality Centre for mine design at the University of Pretoria in South Africa is home to both a 3D stereoscopic theatre and a 3D, 360° cylinder theatre.

The 3D simulated images create authentic mining conditions and scenarios such as underground rock falls that immerse users in the mining experience, as well as providing students with a better understanding of how VR technologies can improve performance and health and safety standards.

VR can also slow down a blast sequence to one-millionth of its real-time speed, allowing engineers to improve both blast and drill rates - and thus mining yields - by simulating different patterns.

Child's play: Minecraft, Occulus Rift and Microsoft HoloLens

Rio Tinto has, since 2008, pioneered innovation as part of its 'Mine of the Future' programme and has a long track record of harnessing technology to create value in the diamond industry including X-ray sorting machines, as well laser technology used to prepare diamonds for cutting and polishing.

The mining multinational has now partnered with New York-based Bravo Media to custom design the Oculus Rift experience, an immersive, computer-generated environment that allows the user to fly above the coast of Canada before taking an interactive journey down the Diavik diamond mine.

Officially released in April, the customisable Oculus Rift VR headset uses state-of-the-art displays and optics designed specifically for VR to provide visual fidelity and an immersive, wide field of view.

Rift's advanced display technology combined with its precise, low-latency tracking system enables the sensation of presence. Its potential for the mining sector covers training and maintenance, as well as enabling companies such as Rio Tinto to animate the resource journey from mine to market.

"Rio Tinto has, since 2008, pioneered innovation as part of its 'Mine of the Future' programme ."

Codelco is spending $3.83bn converting its Chuquicamata mine in Chile and is collaborating with Microsoft on a contest for children to design the best underground mine using the Minecraft computer game.

Microsoft is also working on a version of Minecraft that will use its HoloLens AR headset. HoloLens is the first fully untethered, see-through holographic computer that enables high-definition holograms to come to life − and its potential to revolutionise the mining industry is already being recognised.

"The ability to mix virtual content with reality will provide countless opportunities to improve operational safety and efficiency as well as bringing the corporate offices even closer to the operations on the ground by connecting people in ways never imagined before," states Grey Properjohn of Australian company Vix Technology in a newsletter reproduced in Mining Journal.

"Imagine standing in an underground development heading and becoming visually aware of all the adjacent headings, slopes and declines in the immediate vicinity, or being able to virtualise the trends in ground features like fault-zones or stress-zones... Extend this to drilling from underground and you can virtually eliminate the potential to inadvertently intersect other existing openings.

"The assessment of mining an open pit through underground workings takes on new meaning if you could actually 'see' the workings. Even from an office environment, users can be immersed in the underground operation tracking the locations of all people and assets within the mine in real-time."

Remote interesting: WorksiteVR training and offsite monitoring

Properjohn also acknowledges the role that innovations such as remote operating centres will play in streamlining mining operations, and communicating data and instructions to field personnel.

"Imagine enhancing the efficiency by being able to communicate with people on site, not only using... Skype's built in real-time translations... but actually seeing what they're seeing on-site and interacting with them as they undertake their work," he says. "This could be as simple as providing geological assistance when logging core samples or assisting with specific maintenance operations.

"Microsoft's proof of concept provides remote assistance installing a light switch... the opportunities are limitless considering the remoteness of the operations we're used to dealing with," he adds.

HoloLens also has the potential to help resource geologists and engineers improve their models and plans by enabling them visualise the block model beneath their feet to see the latest grade control drilling results. The technology could also impact traditional borehole logs and core image analysis.

"Rather than simply viewing a 2D representation of logs - or even a 3D view - on a monitor or paper, think about being able to physically 'lift' the core from ground and see the geology, geophysics and imagery overlaid on a 3D representation of the hole in its physical location, with the relative geology of all other boreholes in the vicinity," Paperjohn enthuses. "The potential is amazing and set to revolutionise, if not redefine, current thinking."

One gets the feeling L Frank Baum would approve.