Virtual and Augmented Reality in Medical Imaging

A quick overview of the current uses and future directions of virtual reality and augmented reality in radiology

Virtual reality (VR) and augmented reality (AR), which are commercially accessible for the entertainment and gaming industries (and quite fun!), are coming to medicine, and to medical imaging and radiology in particular. Just as a quick refresher: virtual reality (VR) is a full and isolated virtual display, whereas augmented reality (AR) is the integration of virtual components onto the background of reality (think Pokémon GO). In a nutshell, recent advances in virtual and augmented reality technologies now allow for radiographic data to be used to build a display that has greater integration and interactivity. This has led radiology departments to begin investigating their use as aids in radiology trainee education and clinical care. In this article, we’ll cover just how VR and AR are currently being used in medical imaging and the benefits it offers medical professionals and patients, as well as identify some of the limitations and disadvantages of its use in the field.

• Virtual Reality in Medical Imaging and Radiology
  • Ways that VR is changing Medical Imaging and Radiology

• Augmented Reality in Medical Imaging
• Limitations of AR and VR in medical imaging
• Conclusion

---

Virtual Reality in Medical Imaging and Radiology

As we mentioned above, VR is a computer-generated simulation of a close-to-real-life experience made with an interactive three-dimensional (3D) picture or environment. VR deceives the user’s senses by removing distractions from the outside world, resulting in an immersive experience. To generate a 3D world, virtual reality employs fully enclosed headgear. Interactions inside this virtual environment are made possible by specialized technological equipment, such as sensor-equipped gloves and helmets with internal displays.

In radiology departments, VR can create a virtual representation of a patient’s anatomy to be used for pre-procedural planning, which can influence how a patient’s operation is carried out. Radiologists and radiology students can navigate the 3D environment by walking about with external camera tracking systems or moving their heads while wearing head-mounted display tracking systems. Handheld devices with haptic feedback and vocal gestures can also be used to interact with the environment.

Ways that VR is changing medical imaging and radiology

• VR-Based Images for Radiology Students — Virtual reality is proving to be an excellent tool for radiology students. Radiology students, residents, and radiologists may see 3D scans in real-time using VR-based applications. They may also interact with the image and alter it to enable viewing from a variety of angles.
• VR-Based Teaching Applications provide interactive lectures, with students seeing interventional procedure suites via stereoscopic viewers linked to their smartphones. These interactive lectures are intended to present students with a scenario and then ask them a range of management questions. Common management question focus on:
  • Diagnosis
  • Indications
  • Contraindications
  • Type of sedation
  • Type of equipment
• Quick Response Code (QR) — The learner scans a Quick Response (QR) code after answering the questions. The student is sent to a virtual interventional radiology suite by scanning this QR code. When the student enters the virtual location, he or she may look around and pick what sort of equipment to employ for the hypothetical scenario.
• How and Where Radiologists View Diagnostic Images — Virtual reality has the potential to alter how and where radiologists see images. As the quality of VR pictures improves, 3D imaging may become routine practice, giving radiologists more freedom by eliminating the need for a workstation. Radiologists may examine virtual reality pictures using VR goggles, which would allow them to see scans from practically any location.
• To Help Ease Patient Anxiety — The virtual reality representation of a patient’s anatomy can be used for pre-procedural planning, influencing how a patient’s operation is carried out. A virtual reality simulation of a radiological treatment might make a patient feel more at ease. Through immersive reality, patients can be familiarized with operation suites and recovery rooms, and even with the surgery itself. Immersive reality can be used to mimic a surgery, helping the patient to better understand the phases of the procedure they will undergo and what to expect. This can demystify procedures for patients, which often reduces apprehension and anxiety that stems from uncertainty.

The FDA’s approval of these innovative, 3D virtual-reality technologies for use in the medical field indicates that a paradigm shift is underway, and while the use of virtual reality in medical imaging is still relatively new, it is likely that it will play a significant role in diagnostic radiology soon.

The gist: Virtual reality (VR) in medical imaging is being researched by radiology departments for use in clinical treatment and radiology training. It can also be used for pre-procedural planning, and can allow radiologists and radiology students to navigate the 3D environment

Augmented Reality in Medical Imaging

Augmented reality (AR) is distinct from VR in that the user is not completely immersed in a virtual world, but rather hologram-like entities are superimposed on a real-world background. AR-specific applications have shown promise in a variety of disciplines, with new potential applications in medical imaging emerging. Recent advances in technology have allowed the increased portability of AR hardware, making more widespread utilization possible. The main types of AR devices include see-through head-mounted displays, mobile-based devices, and projection-based devices. Head-mounted displays allow users to perceive a 3D object by passing light from micro projectors onto a waveguide, which is then projected back to the wearer’s eye.

Projection devices were the first incarnation of AR, but because of their total size and mobility, they have had little general adoption (except applications such as portable vein finders). Now, many mobile applications enable smartphones and tablets to serve as a “picture window,” with extra AR information shown on the device’s screen as if it were being viewed by its camera. Still, AR gear is still in its early stages of development, with continuous work focused on resolving its relatively restricted field of vision (now up to 35° with the Microsoft HoloLens), and additional advancements expected in the coming years.

One of applications of augmented reality technologies in medical imaging with the greatest potential is procedure planning and guiding. Here, AR displays are created in the operating room utilizing sophisticated algorithms that translate CT data into a 3D map of structures characterized by density. Depending on the demands of the operation, this map can be shown in a variety of ways, such as on a nearby screen, through a headset, or projected onto the patient. Then, before the first incision, a plan can be sketched into the enhanced surgical field.

Structures such as arteries and tumors can be superimposed into a surgical field display, and organ motions and deformations can be compensated for in real time to refresh the image. This compensates for the loss of tactile feedback during laparoscopic surgeries and provides for more accurate real-time viewing of underlying tissues. Tang et al. (2018) proved the use of these displays for hepatobiliary surgery, but these approaches may be applied to almost any surgical procedure that can be planned ahead of time (AR projectors are already used to incorporate navigation assistance in MRI-guided operations). This effectively reduces radiation exposure for patients while providing surgeons with better planning capabilities.

AR stands to increase the integration of radiographic information in the OR wherever it exists before or during an operation, and radiologists are the gatekeepers of this technology with their understanding of medical informatics and mastery of radiologic anatomy.

Like VR, AR allows for the education of radiology trainees, as well as the engagement of patients. Early AR research revealed gains in student motivation for learning, interaction, and subject learning, because the information is placed in a real-world context, the success of AR for radiology education is considered to be connected to the unique experience and the retained ability for interaction.

The gist: Procedure planning and guidance is one of the most promising uses of AR technology. In the OR, AR can overlay objects such as arteries and tumors onto a surgical field display and organ movements and deformations can be corrected for in real time to refresh the image. AR also enables both the teaching of trainees and the involvement of patients

---

Limitations of AR and VR in medical imaging

Although virtual and augmented reality have enormous promise in medical imaging, there are existing limitations, such as ergonomic limits from continuous usage, relatively high prices for adoption and use, and restricted material availability. Prolonged usage of head-mounted displays, for example, has been linked to neck discomfort, nausea, and vertigo due to latency. When compared to textbooks and currently accessible internet resources, the cost of VR and AR technologies for educational purposes is rather expensive. Costs may be split into two categories: costs for creating and purchasing content and costs for consuming information. Purchasing services from third-party firms is another option, as creating content in-house may be difficult. Because VR and AR are still in their infancy, instructional content is scarce. However, this is likely to change in the coming years.

The gist: While VR and AR have tremendous potential in radiology, there are several limitations that must be addressed, such as ergonomic difficulties from continuous usage, relatively high adoption and use costs, limited availability of material, and the difficulty of generating content

---

Conclusion

AR and VR are the waves of the future in technology-enhanced medicine. Using projectors or headsets to replicate interior structures or peep into a procedure in VR may seem very futuristic, but that reality is closer than ever. And radiology departments already have numerous uses for this budding technology, from a plethora of ways to make use of radiographic displays in patient treatment and the advantage AR displays offer for non-invasive operation. Despite its current shortcomings, the sheer value of AR annotation in procedural planning and the real-time display of radiologic information will propel the technology forward.

With current applications such as patient and trainee education, as well as procedural planning and guidance, the future directions that augmented and virtual reality can take medicine and medical imaging in are as diverse as remote-assisted home treatments for patients and VR curriculums over specific chronic diseases for patients and students alike. The technology has the potential to interact with and enhance many areas of daily medical operations and medical education, and radiologists are in a unique position to lead the way.

---

References

• Andersen, D., Popescu, V., Cabrera, M.E., Shanghavi, A., Gomez, G., Marley, S., Mullis, B. & Wachs, J. (2015). Virtual annotations of the surgical field through an augmented reality transparent display. The Visual Computer, 32(11), 1481-1498.
• Makary, M.S. (2020, May 14). Augmented and virtual reality—Radiology at the center of medical technology. Diagnostic Imaging. https://www.diagnosticimaging.com
• Shah, S. (2019, October 16). Top 4 technologies in medical imaging. Imaging Technology News. https://www.itnonline.com
• Tang, R., Ma, L. F., Rong, Z. X., Li, M. D., Zeng, J. P., Wang, X. D., Liao, H. E., & Dong, J. H. (2018). Augmented reality technology for preoperative planning and intraoperative navigation during hepatobiliary surgery: A review of current methods. Hepatobiliary & Pancreatic Diseases International, 17(2), 101–112. https://doi.org
• Uppot, R.N., Laguna, B., McCarthy, C.J., De Novi, G., Phelps, A., Siegel, E., & Courtier, J. (2019). Implementing virtual and augmented reality tools for radiology education and training, communication, and clinical care. Radiology, 291(3), 570-580. https://doi.org

Disclaimer: The information provided on this website is intended to provide useful information to radiologic technologists. This information should not replace information provided by state, federal, or professional regulatory and authoritative bodies in the radiological technology industry. While Medical Professionals strives to always provide up-to-date and accurate information, laws, regulations, statutes, rules, and requirements may vary from one state to another and may change. Use of this information is entirely voluntary, and users should always refer to official regulatory bodies before acting on information. Users assume the entire risk as to the results of using the information provided, and in no event shall Medical Professionals be held liable for any direct, consequential, incidental or indirect damages suffered in the course of using the information provided. Medical Professionals hereby disclaims any responsibility for the consequences of any action(s) taken by any user as a result of using the information provided. Users hereby agree not to take action against, or seek to hold, or hold liable, Medical Professionals for the user’s use of the information provided.