Instructive head-mounted display system: pointing device using a vision-based finger tracking technique applied to surgical education

Introduction

Recent technological advancements have led to progress in operative procedures, especially minimally invasive surgery. The da Vinci Surgical System has dramatically transformed endoscopic procedures, which were previously carried out with operators observing a flat panel display [1–3]. The da Vinci Surgical System provides the operator with a three-dimensional view through the immersive viewer at the surgeon console. With this system, the operator sits on the console with his or her head tilted forward, so that the operator is secluded in the console and isolated from the other participants attending the surgery. Another novel image monitoring system, the personal head-mounted display (HMD) system, has also emerged, providing spectacular improvement in terms of resolution, wearability, and weight. The application of the HMD system has been reported in ultrasound scanning, anesthesia management, endourological surgery, laparoscopic surgery, and gasless single-port urologic surgery [4–10]. The modern HMD, designed for medical use, is now commercially available; the HMM-3000MT (Sony Corporation, Tokyo, Japan) is a 490-g binocular HMD composed of 0.7-inch organic light-emitting diode screens (resolution: 1,280 × 720 pixels) for each eye.
The HMD-based monitoring system is distinct from that of the da Vinci Surgical System. The HMD system enables image sharing among all HMD wearers including the operator, assistants, anesthesiologist, and even students and patients. Using the imaging splitter, the images can be delivered to multiple HMDs, as many as needed, depending on the number of participants in the procedure. However, owing to the sense of immersion, communication between participants can be reduced with this system. Based on the idea of providing perceptual support to the HMD wearer, we reported on the use of a video camera mounted on the HMD to support the operator’s spatial perception by supplying a comprehensive panoramic view [8]. We further think that the ability of the operator wearing the HMD to use a pointer on the endoscopic view without touching any pointing device would benefit communication.

Aim

To support interactive intraoperative communication among HMD wearers, we integrated vision-based finger tracking into our system as a pointing device. In the current paper, we report a case in which an operator communicated with medical students using this system during transurethral resection for bladder cancer.

Material and methods

Our vision-based finger tracking system is composed of a commercially available real-time video camera, which is mounted on the HMM-3000MT, and computer software that enables tracking of the tip of the index finger and superimposing the marker on the endoscopic view. Photo 1 shows an overview of the system. The software converts an HMD-mounted video image into a binary image based on the green color in order to detect and track the green surgical globe in real time. A sequence of points contouring the operator’s hand is created based on the tracked data. The degree between three adjacent points of the contour is calculated, and the position of the tip of the index finger showing a low angle (less than 60°) is marked as a circle. The position of the circle is superimposed on the endoscopic view in real time. The time gap between the endoscopic view and the superimposed pointer is around 0.2 s.
We used this system in an experimental demonstration on a 54-year-old man who underwent transurethral resection for bladder cancer. The operator and the medical students each wore an HMD in which the endoscopic view was displayed, and the video camera was mounted on the operator’s HMD (Photo 2). The position of the marker was superimposed on the endoscopic view. The superimposed image and the video camera view were displayed in the HMD using the picture-in-picture technique, which was simultaneously shared with the operator and the medical students. During the operation, the operator used this finger-tracking pointer to explain the intraoperative findings to the medical students.

Discussion

As a novel personalized imaging monitoring system, the HMD system provides the HMD wearer with a sharp high-contrast view in an ergonomically natural position. In contrast to the da Vinci Surgical System, the HMD system is compact, easily introduced, and affordable. Importantly, various types of devices can be connected to the HMD system. The HMD system can be easily customized and applied to many types of medical procedures including ultra-scanning, esophagogastroduodenoscopy, and bronchoscopy, even outside the operating room. The HMD has enormous potential to dramatically alter medical practice.
The potential of the HMD system in the clinical setting has been further evaluated. The HMD system can be used not only as a personalized image monitoring device, but also as a wearable device for multisensory visualization. In the current paper, we experimentally used a finger tracking system-based pointing device integrated into the HMD system. Using this system, the operator can communicate visually while maintaining hand sterility. This system can be potentially useful for interactive instruction and communication with other attending physicians and medical students.
Many motion-capture tracking technologies are used for 3D design and animation [11–13]. Here, we selected the vision-based finger tracking technique because of its low cost and the compact size of the system. Furthermore, this technology does not require initial registration. However, the time gap between the endoscopic image and the movement of the superimposed pointer may represent a drawback of this system. Further studies are needed to improve and widen the application of the HMD system for medical use.

References

1. Binder J, Bräutigam R, Jonas D, Bentas W. Robotic surgery in urology: fact or fantasy? BJU Int 2004; 94: 1183-7.
2. Witkiewicz W, Zawadzki M, Rząca M, et al. Robot-assisted right colectomy: surgical technique and review of the literature. Videosurgery Miniinv 2013; 8: 253-7.
3. Czudek S. Robotic surgery – a taste of Hollywood? Videosurgery Miniinv 2013; 8: 95-8.
4. Liu D, Jenkins SA, Sanderson PM, et al. Monitoring with head-mounted displays in general anesthesia: a clinical evaluation in the operating room. Anesth Analg 2010; 110: 1032-8.
5. Liu D, Jenkins SA, Sanderson PM, et al. Monitoring with head-mounted displays: performance and safety in a full-scale simulator and part-task trainer. Anesth Analg 2009; 109: 1135-46.
6. Prendergast CJ, Ryder BA, Abodeely A, et al. Surgical performance with head-mounted displays in laparoscopic surgery. J Laparoendosc Adv Surg Tech A 2009; 19 Suppl 1: S237-40.
7. Kihara K, Fujii Y, Masuda H, et al. New three-dimensional head-mounted display system, TMDU-S-3D system, for minimally invasive surgery application: procedures for gasless single-port radical nephrectomy. Int J Urol 2012; 19: 886-9.
8. Yoshida S, Kihara K, Takeshita H, Fujii Y. A head-mounted display-based personal integrated-image monitoring system for transurethral resection of the prostate. Videosurgery Miniinv in press.
9. Yoshida S, Kihara K, Takeshita H, et al. Head-mounted display for personal integrated-image monitoring system: ureteral stent placement. Urol Int in press.
10. Takeshita H, Kihara K, Yoshida S, et al. Clinical application of a modern high-definition head-mounted display in ultrasonic examination. J Ultrasound Med in press.
11. Roy S, Bryant JL, Cao Y, Heck DH. High-precision, three-dimensional tracking of mouse whisker movements with optical motion capture technology. Front Behav Neurosci 2011; 5: 27.
12. Šenk M, Chèze L. A new method for motion capture of the scapula using an optoelectronic tracking device: a feasibility study. Comput Methods Biomech Biomed Engin 2010; 13: 397-401.
13. Zelman I, Galun M, Akselrod-Ballin A, et al. Nearly automatic motion capture system for tracking octopus arm movements in 3D space. J Neurosci Methods 2009; 182: 97-109.

Received: 9.10.2013, accepted: 23.01.2014.