Transforming Reality: How Gaming Gear is Impacting Health Care 

Thirty years ago, virtual reality looked more like science fiction than science. Headsets were oversized, graphics were pixelated and the applications left more to the imagination than the experiences themselves.  

Early users waited in long lines at arcades to strap on a headset that promised an escape into another world, only to be greeted by blocky landscapes and lagging movements that broke the illusion before it even began. It was fascinating, but it was far from transformative. 

YouTube Video bOoO7WtPVcY

Fast forward to today, and VR is almost unrecognizable from those humble beginnings. Headsets are lightweight, wireless and powerful enough to create environments that seem so real that the brain forgets they aren’t. Entire industries have embraced VR as a tool to push boundaries; engineers design and test machinery in immersive digital spaces while soldiers train for missions in environments that replicate real-world challenges without real-world risks.  

With the increase in quality and decrease in costs, VR is pushing its way into applications its early creators could only dream of—and health care is not exempt from this transformation. At Duke University, close interdisciplinary ties between medicine, engineering, graphic design, visual arts and nursing are bringing this quickly evolving technology into the spotlight. 

“The interdisciplinary strengths of Duke make it a perfect place to explore new uses for these kinds of emerging technologies,” said Nanthia Suthana, professor of neurosurgery and biomedical engineering at Duke. “My lab works with students of game design, electrical and biomedical engineering, neuroscience, psychology, and even visual arts. It’s unusual for a neuroscience lab, but it presents powerful opportunities.” 

After months of research and discovery performed in the ICU by Ashley Frith, a researcher who met Gorlatova at a Georgia Tech speaking engagement and asked to join the project, the team had a solid idea of what an interactive VR game for physical therapy in an ICU might look like. To turn that idea into a reality, they engaged Sarah Eom, a PhD student working in Gorlatova’s lab, and students from Duke’s Master of Engineering of Game Development, Design & Innovation program, Ivory Xu and Liheng Zou. 

“Ashley was truly a godsend, we wouldn’t have made nearly as much progress as fast as we did without her,” said Granger. “She spent a lot of time in the ICU with Anna and our research team interviewing patients and shadowing clinicians to truly understand what was needed.” 

A couple of semesters of further refinement and development later, the team now has a functioning VR experience built to engage and challenge patients recovering in the ICU. It’s not entirely unlike the popular VR game “Beat Saber,” in which gamers swing virtual Star Wars-inspired light sabers at oncoming boxes to the beat of hit songs. Given the extreme limitations of these patients, however, the Duke team’s game is tailored to patients in ICU. Rather than using controllers that may be difficult to hold, the VR system tracks patients’ hand movements. And a setup stage conducted by clinicians ensures that the game matches the user’s movement abilities and surroundings filled with lines, tubes, and other life support equipment.   

Then the flying woodland fairy arrives. 

Like a large butterfly with a spherical body and two stubby arms, the fairy-inspired guide offers instructions and encouragement as patients interact with the game. Two strings of colored boxes approach the player from the ocean’s horizon. As they get close enough, players move their virtual hands to stay in contact with each slowly moving chain. As they move forward in the game, the levels slowly increase in difficulty while the program keeps track of their performance. 

“Lying in bed and moving your arms around isn’t a big deal for you and me, but these patients have just had major surgery, so their body doesn’t work the way it normally would,” explained Anna Mall, a clinical nurse specialist in the Duke Heart Center, who led the clinical side of the project’s development. “And these rooms have tons of equipment in them, so fitting enough people in to help with physical therapy can be challenging. This type of a VR game can definitely help with that.” 

Decrypting the Complex Puzzles Underlying Neural Disorders

For many patients suffering from debilitating neural disorders such as post-traumatic stress syndrome (PTSD), the slightest trigger in an otherwise calm and controlled setting can set off an episode. The surrounding environment doesn’t even need to be that similar to the experiences that caused the trauma. 

It should come as no surprise, then, that the increasingly realistic worlds created by VR technology can also trigger stress responses. While some would see this as no more than a potential liability requiring pages of disclaimers, Nanthia Suthana saw this capability as an opportunity.

PTSD, compulsive behaviors, eating disorders and other neurologically based disorders all involve a complex set of physical and neural actions and responses. Palms sweat. Eyes dart about frantically. Heart rates increase. Specific regions of the brain light up. By simultaneously tracking these physiological reactions during an event, Suthana hopes to discover better ways to treat them. 

“When we can simulate these experiences and these behavioral and cognitive states, we can better understand the brain’s activity during these moments and develop therapies that can intervene,” said Suthana, professor of neurosurgery and biomedical engineering at Duke.  

Neural activity deep within the brain, however, is not easy to monitor or measure. In animal studies, electrodes are implanted in specific regions of the brain to record responses to stimuli and physical behavior. The same could be done with humans, but any researcher would be hard pressed to find enough people willing to get brain implants for the study. 

Suthana found a way around this hiccup by recruiting patients who already have them. 

Deep brain stimulation is an increasingly common method for combatting the worst symptoms of neural diseases that affect people’s motor skills such as Parkinson’s disease and essential tremor. It is estimated that more than 160,000 people worldwide have had this technology implanted into their brains, with 12,000 more taking place each year. Odds are pretty good that at least a few of them also have PTSD. 

“But we can’t record these people’s brain activity while they’re living their daily lives, so we ask them to volunteer to come to our lab,” Suthana said. “They have a very unique window into these brain areas that lets us see how neural activity changes during these various experiences.” 

In Suthana’s laboratory—a large, wide-open basement space that was a retail store in its former life—volunteers with these implants walk through VR environments designed to trigger a stress response. For example, participants might walk through the aisles of a small bodega when, suddenly, a large virtual spider jumps out of nowhere and on to their face. 

When it does, Suthana’s team’s custom-built wireless interface records brain activity to discover patterns that immediately lead to an episode. But they don’t stop there. Participants are also decked out in a wide range of monitoring devices so that every aspect of their response can be recorded and analyzed.  

Caps fitted with electroencephalogram (EEG) sensors capture surface-level brain activity. Cameras within the VR headset track eye movement. Gloves monitor levels of palm moisture. A body suit decorated with strategically placed white balls helps arrays of video cameras mounted on the walls track their every move. 

“We have all these cameras that can measure their movements, and we can study all of these things together to try to get a picture as to what’s going on,” Suthana said.

With such promising results already in hand, and with Duke’s vast array of interdisciplinary resources, Suthana is excited to see what more they can accomplish in the coming years. They are already working with Diego Molina, a student in the Master of Engineering of Game Design & Development, to better tailor bespoke VR experiences to specific volunteers. And the Duke University School of Medicine has been an obvious partner for finding volunteers and collaborating in the analysis of the data. 

“This kind of work requires clinicians, neuroscientists, neurologists, neurosurgeons, psychiatrists and great engineers,” she said. “There are very few places that have all those things. Duke not only has them, but it has multiple top people in all those areas: the best engineers, the best surgeons, the best neuroscientists. And they all want to work together! That creates a community that is rare to find.” 

“The core of HoloSNS is the ability to overlay patient-specific brain activity data from multiple sources onto anatomical imaging sets,” McIntyre explained. “It’s been a slow grind because it’s really engineering-intensive to make it work in a clinically relevant environment. But when it does work, it’s awesome, and the clinicians absolutely love it because it gives them a new way of seeing things.” 

For example, multiple neurosurgeons from disparate corners of the globe can look at a patient’s brain scans together in 3D to help plan complex surgeries.  

“We’ve even made some important clinical decisions based on the technology that would have been very difficult, if not impossible, to do with regular navigation technologies,” McIntyre added. 

While there have been successes, McIntyre says that the technology is still a long ways away from “prime time clinical use.” But there are other important use cases, he says, that are much more manageable to achieve in the near future. 

“Maybe the biggest market for this technology is actually in patient education or patient preparedness for whatever procedure they might undergo,” he said. “It’s very difficult to explain these very complicated surgeries to patients. They just don’t have any context for even understanding what they might be undergoing. A computer screen helps, but it’s something different to see it in 3D and see it in your own MRI data. So I see a lot of potential for that.”