A research partnership between BMW and Meta is analysing how car-based virtual reality could be implemented in the future.
One of our core values here at GMW Perth is Past & Present – appreciating and respecting the past, while continuing to innovate and stay up to date with industry news and trends. We recently found some interesting research by BMW and Meta surrounding the technological advancements in the automotive industry. We explore that research here in this blog.
Origins of the BMW x Meta Partnership
Meta, the parent company of social media giant Facebook, has been working diligently to make augmented reality (AR), artificial intelligence (AI) assisted smart glasses happen as part of Project Aria. The camera and sensor studded Project Aria glasses are being tested in a wider range of environments, including cars.
A partnership with BMW sees Meta exploring the impact of driving while wearing smart glasses – an area that has raised concerns in the past with technology including Facebook’s Ray-Ban Stories glasses and Google’s Google Glass headset. The partnership aims to investigate how both augmented and virtual reality (VR) could be integrated with modern vehicles to enhance the passenger experience and how AR glasses could be used effectively in a moving vehicle.
Manufacturers of AR glasses explain that they are intended for all day wear, but a key part of our lives is driving or riding in moving vehicles. The partnership between Meta and BMW seeks to understand how these technological advancements can provide a unique and valuable experience to consumers.
What Is AR & VR?
You may have heard of the terms AR and VR before, but do you know the difference between them? They are often confused with one another, so let’s clarify the differences.
Augmented Reality (AR)
Augmented reality (AR) involves overlaying visual, auditory or alternative sensory information onto the real world to enhance a user’s experience. These digital elements are delivered via technology and it is a growing trend amongst companies, particularly those involved with business applications and mobile computing. A primary goal of AR is to highlight specific features within the physical world, increase understanding of those features and derive insights that can be applied.
Virtual Reality (VR)
Virtual reality (VR) is a simulated three-dimensional environment enabling users to interact and explore in a way that replicates reality, through the senses of the user. The environment is created with computer hardware and software, and often requires users to wear devices such as a helmet or goggles. The deeper a user can immerse themselves in the environment, blocking out their surroundings, the more they are able to accept it as real.
The Research
What if the time we spend in cars could be more social, more productive and more entertaining than ever before?
This is the question that forms the basis of the research partnership between BMW and Meta. Announced in 2021, the project explores how AR and VR could be safely integrated into smart vehicles to enhance the passenger experience. As an industry leader in cutting-edge automotive technology, BMW has been experimenting with the possibilities of AR integration for over a decade. A natural progression of this research was a joining of forces between BMW and Meta’s Reality Labs Research, working together to level up the future of travel.
At first glance, the problems associated with delivering high quality AR and VR content for passengers may not be obvious – particularly with the advancements of technology in this space. Meta’s Oculus Insight technology, commercially released in 2019, delivered a tracking system that accommodates a person’s full range of movements (known as six degrees of freedom or 6DOF), while also pinpointing the location of handheld controllers, and the headset with extreme accuracy in real time.
Moving vehicles pose a tricky challenge, as tracking technologies like Oculus Insight use both cameras and inertial motion sensors (IMUs) to estimate the location and motion of the headset. In a moving environment these two data inputs are in conflict – the cameras observe motion relative to the car’s interior, whereas the IMUs measure acceleration and rotational velocity relative to the world. VR headsets fail to display virtual content that is stable from within a vehicle as it turns and accelerates.
As a work-around for this issue, Meta incorporated IMU data from BMW vehicles’ sensor array in real-time to the tracking system of the Project Aria research glasses. This additional layer of information allowed the system to calculate the position of the glasses relative to the car – a major breakthrough which enabled compelling virtual and mixed reality experiences in moving vehicles. The next step of the research will be to add the location of the car relative to the world, allowing for world-locked rendering (the placement of digital assets with reference to physical objects).
The research conducted by BMW and Meta demonstrates that with additional IMU measurements from the vehicle, stable, car-locked virtual and mixed reality content can be displayed to passengers while driving – even when the vehicle is making rapid turns, accelerating or travelling on uneven roads.
Potential Benefits
In time, AR is expected to help users put the handheld devices away and interact more naturally with both the digital world, and the real world – an interface that is intuitive and can proactively help meet people’s needs rather than react to user input. Meta believes that AR glasses could eventually assist passengers with navigation.
From the research discussed above, BMW and Meta engineers were able to attain stable gaming, productivity, entertainment and meditation experiences within a moving BMW vehicle. Claus Dorrer, head of BMW’s Technology Office in California, envisions a range of use cases for AR devices incorporated into vehicles. These include alerting vehicle occupants to hazards in their surroundings, displaying information on the vehicle’s condition and helping the user find the vehicle in a crowded car park.
Final Thoughts
With technological advancements moving so rapidly in the current day, it is a complete possibility that AR and VR experience becomes a normalised aspect of vehicle travel. With that being said, the primary focus for all parties involved should be the safety of occupants within the car and pedestrians, who will both adopt and reject the use of the technology.
The future of automotive technology is incredibly intriguing and we may only be scratching the surface of possibility.
At GMW Perth, one of our core values is past and present – appreciating and respecting the past, while continuously striving to innovate and keep a finger on the pulse of industry. To learn more about our core values, visit the link here.