Exploring virtual reality and artificial intelligence through the healthcare industry

Amidst engagements with a healthcare client at Cognizant, we delved into infusing AI through digital avatars in a virtual environment with the goal of simulating patients with Type 1 Diabetes. The learnings of that exploration and research have helped visualize the many ways AI could be used in the future when converging into the virtual or mixed reality world.

Research entailed a primary focus on understanding the role AI will take when merging into virtual reality (VR) and the many improvements it could bring to a technology that, despite being promising and exciting, has struggled to go from the curve of early adopters into a useful and well-adopted product over the past few years.  

Throughout user interviews within the project, the struggles users faced started to become clear. Within the virtual world, users experienced a multitude of frustrations, from nausea during the navigation and walking, to simply the inability to connect and interact with other peers resulting in a lonely experience where the focus initially was collaboration within the metaverse.

Subsequently, we added an AI character into the experience which helped creating amusement and improved the user experience due to being a companion in the virtual world. In short: artificial intelligence has added a myriad of value to the use of a metaverse experience.

Healthcare is already one of the areas with most visible advantages of using virtual reality, whether it is the ability to explore microscopical forms from inside a body or to re-create activities that once used complex tools and live organic matter to be studied. The current technologies embedded within VR are already being adopted widely, but with the exponential growth and use of large language models (LLMs) and generative AI, these industries are going for an even higher stake when it comes to innovation.

In a recent study by the University of Central Florida, they were able to highlight different areas where it is possible to see a growth within the AI academic community anytime there was an overlap on the use of mixed reality. The research shows that medical training is at the top of the list of research topics currently in development, showcasing that healthcare is front and center when it comes to exploring the combined power of virtual reality and artificial intelligence.

Image by William Terra [source]
The Combination of Artificial Intelligence and Extended Reality: A Systematic Review, University of Central Florida, 07 Sept 2022.

The different forms in which AI and virtual/mixed reality have been combined to improve user experience are endless. It only depends on your team’s requirements and how much complexity can be added to your project. During our current explorations in healthcare, tools like Luma.AI genie can be leveraged to quickly generate 3D organs that are then exported as models for prototype exploration, allowing the creation of mid- to high-fidelity 3D models using a text-to-3D-model tool.

Artificial intelligence’s knack for deciphering complex patterns and making accurate decisions is revolutionizing medical diagnostics. By analyzing medical images, AI algorithms can spot anomalies that might escape the human eye, offering more precise diagnoses and enabling early interventions.

But it does not stop there. Imagine a surgeon using augmented reality (AR) glasses that overlay critical patient information in real time during a procedure. With this invaluable data at their fingertips, surgical accuracy is elevated to new heights, minimizing risks, and improving patient outcomes.

Image by Nidia Dias as part of the Visualizing AI project launched by Google DeepMind

Challenges and ethical considerations

The fusion of AI and VR, while exciting, also presents challenges and ethical questions. The more realistic and personalized VR experiences become, the more data they require — often sensitive information about users’ behaviors, preferences, and reactions. Ensuring this data’s security and privacy is a significant challenge. In areas like pharmaceutics and healthcare, where information is controlled and secured by international agents (such as the IGJ in the Netherlands), it can become a challenge that must be considered on any project in this area.

Moreover, as AI makes VR experiences more lifelike and immersive, we must consider the implications for our perception of reality. How do we ensure that the line between the virtual and the real doesn’t become dangerously blurred? Among the issues with data privacy, algorithmic bias, and potential addiction to immersive experiences, we must safeguard this technology with transparent guidelines and set regulations to ensure that AI and MR are developed and used ethically, respecting both individual rights and social well-being.

Looking further into the future collaboration of AI/VR, here is the view of Pablo Ferreras, Service and Interaction Designer and GenAI Program Co-Leader at Cognizant Netherlands:

“In the medium term, I see AI being used to fill these virtual worlds with artificial life in the same way that, right now, GenAI is being used to embody simpler forms of digital interaction such as website chatbots or search engines. In the long term it might create worlds for games and movies completely on its own, just driven by user preferences and data.”

Overall, AI is enhancing the realism of VR and AR experiences by enabling more natural language interactions, generating accurate 3D models of real-world objects and creating more dynamic and interactive experiences. While there are challenges to address and ethical considerations to bear in mind, the promise of this powerful technological fusion presents many exciting opportunities that could reshape personalization in virtual experiences to a whole new level. The question is: will LLMs and neural networks bring the boost that is needed for the popularization of mixed and virtual reality technology?