Introduction
High-resolution endoscopes and surgical navigation systems have been developed to assist in minimally invasive surgery. In laparoscopic surgery, however, the surgeon performs all surgical procedures based on information shown on the display. Ideally, the lead surgeon manages the field of view necessary for the operation, but in laparoscopic surgery it is the assistant who manipulates the laparoscope and manages the field of view, and participation of the assistant is therefore mandatory. It is difficult to achieve perfect coordination between the surgeon and the assistant using only verbal instructions, and the assistant’s inability to produce the images intended by the surgeon often interferes with the operation.
Recent advances in eye tracking technology have enabled accurate detection of gaze at a medium distance [1, 2]. This technology can detect the exact location of the surgeon’s gaze point in the operation room, and function as a user interface for the surgeon to select the required information in the clean field during surgery [3–5].
Aim
We have developed a touchless user interface that employs an infrared-light-based eye-tracking device fixed to a flat-panel display. Information acquired regarding the surgeon’s gaze point enables image size to be digitally zoomed according to the surgeon’s voice command, and a magnified image from the endoscope is displayed accordingly. The development of this system and our trial experience in application to laparoendoscopic single-port partial nephrectomy has been reported previously [5].
Here, we report the ability of the endoscopic image processing system to manipulate laparoscopic images according to the surgeon’s gaze point and voice command in laparoscopic percutaneous extraperitoneal closure (LPEC) of inguinal hernia in children.
Material and methods
This study was approved by the Institutional Review Board of our institution (approval No. M2018-276) and written informed consent was obtained from each participant.
The system uses an endoscopic video image size modulator that we developed previously, consisting of an eye-tracking device and computer software that centers on the acquired gaze point of the surgeon, and digitally zooms the images for display in response to voice commands.
An overview of the system is shown in Photo 1 [5]. We used the Tobii Eye Tracker 4C system (Tobii Technology AB, Danderyd, Sweden). This infrared-based device calculates the direction of the point of gaze using the vector between the pupil center and the corneal reflection produced by the device. The eye tracking device was fixed to a flat panel display and the surgeon’s gaze point on the display was acquired. The video image is processed by the computer software according to the voice command and displayed on the flat panel display in real time.
Photo 1
Steps taken by the developed computer software system to obtain a digitally zoomed image for display, centered on the surgeon’s gaze point
First, we analyzed the feasibility of the system, using a LPEC surgical trainer in a preclinical setting. Four pediatric surgeons with 8 to 21 years of experience completed a preclinical trial of the system. A video camera of resolution 4K was fixed at a distance of 90 cm from the flat panel display, with the eye tracker attached. Using the developed endoscopic image processing system, the video image of the LPEC operation field shown on the display was adjusted to show a digitally zoomed view of the upper right corner of the screen (corresponding to the LPEC surgical operation field) by gazing at the intended zoom center and by voice commands. After returning to the equal magnification display image, the LPEC operation field at the upper left corner of the screen was digitally zoomed, and then returned to equal magnification with the original video image. An anonymous 5-point Likert scale questionnaire was used to evaluate the ease of adjusting the displayed endoscopic images as intended, smoothness of image movement, image quality after magnification, and discomfort and time lag associated with image processing.
As a clinical study, we applied this system to LPEC for pediatric inguinal hernia, performed by the same four surgeons as in the preclinical trial. The surgeon stood on the patient’s left and the display was placed at the foot side. In this setting, the eye tracking distance was approximately 90 cm. We used a 5 mm 30-degree rigid endoscope with 4K resolution. All surgical procedures were performed using Takehara’s technique [6]. A 5 mm port was placed at the umbilicus, though which a laparoscopic scope was inserted. After placement of a 3 mm forceps trocar in the left lower abdomen under endoscopy and observation of the bilateral inguinal regions, the endoscope was fixed with a holder towards the internal inguinal ring of the side to be operated on. The direction and insertion depth of the scope were adjusted so that the lateral side of the internal inguinal ring and medial side of the internal umbilical folds could be seen in the same field of view. After the endoscope was fixed, the procedure was basically performed by the surgeon alone, including needle manipulation. To display the appropriate operation field, the surgeon adjusted the site and size of the image displayed on the monitor under control of their gaze point and voice instructions (Photo 2) (Supplementary Material 1). Wound closure was performed using a Lapaherclosure needle and 2-0 non-absorbable suture thread. Within 1 week after surgery, pediatric surgeons who had participated in the surgery were asked to evaluate the usefulness of the system via an anonymous 5-point Likert scale questionnaire.
Photo 2
View of the setup. The eye tracking device on the monitor recognizes the surgeon’s point of gaze on the image. The surgeon gives voice commands that are received by a microphone, and the image is enlarged and moved around the gazing point accordingly. The endoscope is fixed in a holder, and no scope manipulation is performed by the assistant
Results
In the preclinical evaluation, all four pediatric surgeons completed the questionnaire. All responded “excellent” or “good” for ease of adjusting the displayed screen as intended, smoothness of image movement, and image quality after digital zooming. None of the participants felt there was a time lag related to image processing.
In the clinical study, LPEC using the system was performed on fifteen patients. The median age of the patients was 2 years (3 months–20 years) and the median weight was 10.8 kg (7–44 kg). Ten (75.0%) patients were boys and 5 (25.0%) were girls. Eleven (73.3%) patients underwent LPEC for unilateral inguinal hernia and the remaining 4 (26.7%) for bilateral inguinal hernia. In 3 (20.0%) cases, LPEC and other techniques were performed simultaneously (umbilical plasty n = 1, orchiopexy lingual dissection n = 1, and urethrostomy n = 1). In cases in which LPEC alone was performed, the median operative time for unilateral inguinal hernias (n = 10) was 55.5 min (47–83 min), and the operative times for bilateral inguinal hernias (n = 2) were 57 and 80 min, respectively.
No patient had complications of Clavien–Dindo classification 1 or higher. In all cases, the surgery could be completed with the scope remaining fixed in position, using eye guidance and voice instructions to magnify and move the displayed endoscopic image (Photo 3) (Supplementary Material 2). The median number of voice commands used per LPEC for one side was 15 (range: 10–22). The displayed image was digitally zoomed up to 2.5× (screen resolution of 1534 × 864 pixels). The surgeon completed all surgical procedures alone, including needle manipulation, except when the patient’s testes were in traction.
Photo 3
Representative monitor images obtained when using the system during LPEC. In all of these images, manipulation of magnification and image movement was performed solely by voice recognition and eye tracking, with the scope fixed. A – The camera is fixed so that the entire operative field, centered on the inner groin ring, is captured in the image. B – The internal inguinal ring is observed by magnifying the image. C – The needle tip is followed while magnifying and moving the image. D – The needle tip punctures the abdominal cavity. E – The image is then zoomed out to include the entire area. F – Movement of the needle tip is followed with magnification and movement of the image. G – Retrieval of the thread in the abdominal cavity. H – Ligation of the internal inguinal ring
All four pediatric surgeons who participated in the surgery completed the questionnaire about the usefulness of the system. All answered “excellent” or “good” in terms of ease of adjustment of the displayed image as intended, and smoothness of image movement and digital zooming. All answered “excellent” or “good” for resolution of the endoscopic image after digital zooming when performing the surgical procedures. No surgeon experienced image delay that would affect the surgical operation. Compared with the situation of the scopist handling the endoscope, all the participants rated the ease of surgical operation as “excellent” or “good.” Reasons for the high ratings included “no camera shake” and “the screen is exactly as intended, allowing me to concentrate on the surgery.” All surgeons considered that applicability of the system for other surgical procedures was “very promising” or “promising.”
Discussion
In this paper, we report our initial experience in applying a touchless user interface to LPEC. A magnified endoscopic image centered on the surgeon’s gaze point on the monitor was displayed as intended in response to eye tracking and voice commands, and the system demonstrated good operability and utility.
This system provides a magnified endoscopic image of the area required by the surgeon, without camera shake, enabling the surgeon to concentrate on the procedure. Like a binocular loupe used in laparotomy, digital magnification of images allows the surgeon to perform more delicate and certain operations. This is an advantage that cannot be obtained by simply using a large, high-resolution monitor. In this study, an assistant was required to assist in opening the abdomen and inserting the port, but in the case of a single-hole procedure, a surgeon using this system could possibly complete the surgery alone. This may make laparoscopic surgery possible even in hospitals that have only one pediatric surgeon due to depopulation of medical staff. The system may also be useful for reducing the burden on the scopist during long operations. The ability of the system to acquire very high quality video of endoscopic surgery, without camera shake, suggests its value for conference presentations and educational purposes. The system was easy to handle and after the second case there was little increase in operative time due to use of the system. The system was effective in all cases, from infants weighing 7 kg to 20-year-old adults, regardless of their body size.
Many studies have reported the application of eye-tracking devices to various laparoscopic surgery systems. Vine et al. reported the effectiveness of a laparoscopic manipulation training program using an eye-tracking device [3]. Bright et al. analyzed the eye movements of skilled and novice surgeons performing transurethral resection of the prostate in virtual reality using an eye tracker and reported differences in gaze strategies [4]. In actual laparoscopic and microscopic surgery, eye-tracking devices used to analyze the surgeon’s gaze points and movement patterns on the monitor revealed clear differences between experts and novices [7–10]. Eye-tracking devices are also expected to be effective for education in laparoscopic surgery. It has been reported that display of the surgeon’s gaze on the monitor has improved communication among surgical staff and learning among residents and medical students [11], and that projection of the supervisor’s gaze on the monitor to guide the surgeon has reduced operative time and the number of errors.
The Senhance Surgical Robotic System (TransEntrix Inc., Morrisville, NC, USA) is a robot-assist surgical system in which the surgeon uses eye movements to control the robotic arm that holds the endoscope and uses head movements to control zoom [12]. Robotic systems that control the movement of the endoscope have the advantage of being able to adjust the position of the endoscope to obtain the desired field of view. In our system, a fixed high-resolution endoscope image is digitally zoomed around the surgeon’s gaze point and the processed endoscope image is displayed as required by the surgeon. Although our system is limited to using information obtained from a fixed endoscope, this compact system is low cost and can be easily applied to a variety of surgical procedures.
As a limitation of the system, digital zooming of the endoscope image results in a decrease in resolution. However, the 4K endoscope used in this trial maintained resolution of 1534 × 864 pixels even for image magnification of up to 2.5×, which had little effect on the surgical operation. Further magnification of endoscopic images will become possible as endoscopes of even higher resolution become available in the future.
The scopist can also use this system to perform endoscopic manipulation. When the endoscope is fixed, however, organs interposed between the endoscope and the operative field may make it difficult to obtain a magnified view of the operative field, and the fixed position of the endoscope must be readjusted. Its usefulness in gastrointestinal surgery, where the endoscopic field of view needs to be adjusted many times, requires further evaluation in the future. The system might be suitable for temporary use to perform procedures with a fixed field of view without camera shake in situations where delicate manipulations are required, such as bile duct jejunostomies and pancreatic duct jejunostomies. In the future, we intend to apply this system to various other surgical procedures to clarify the usefulness of the gaze guidance system in laparoscopic surgery.
