Automated Stand-alone Surgical Safety Evaluation for Laparoscopic Cholecystectomy (LC) using Convolutional Neural Network and Constrained Local Models (CNN-CLM)
Abstract
In this golden age of rapid development surgeons realized that AI could contribute to healthcare in all aspects, especially in surgery. The aim of the study will incorporate the use of Convolutional Neural Network and Constrained Local Models (CNN-CLM) which can make improvement for the assessment of Laparoscopic Cholecystectomy (LC) surgery not only bring opportunities for surgery but also bring challenges on the way forward by using the edge cutting technology. The problem with the current method of surgery is the lack of safety and specific complications and problems associated with safety in each laparoscopic cholecystectomy procedure. When CLM is utilize into CNN models, it is effective at predicting time series tasks like identifying the sequence of events in the Laparoscopic Cholecystectomy (LC). This study will contribute to show the effectiveness of CNN-CLM approach on laparoscopic cholecystectomy, which will frequently focus on surgical computer vision analysis of surgical safety and related applications. The method of study is deep learning based CNN-CLM to better detect nominal safety as well as unsafe practices around the critical view of safety and AI-based grading scale. The general design flow of AI-recognition of surgical safety is firstly collecting safety surgical videos for frame segmenting and phase according to the image context by surgeon reviewer by CNN-CLM. For this advance research, the dataset is splatted into three main parts where 70% of which is used for training, 15% of which is used for testing and the rest for the cross validation, to achieve the accuracy up to 98.79% of this specific research. For result part, different metrics of CNN-CLM to evaluate the performance of the proposed model of safety in surgery. The study uses one of the top three performing methods CNN-CLM for the evaluation yields and anatomical structures in laparoscopic cholecystectomy surgery.
Full Text:
PDFReferences
D. A. Hashimoto, G. Rosman, and D. Rus, “Artificial Intelligence in Surgery: Promises and Perils,” Ann Surg, vol. 268, pp. 70–6, 2018.
B. Le Roy, E. Ozgur, B. Koo, E. Buc, A. Bartoli, “Augmented reality guidance in laparoscopic hepatectomy with deformable semi -automatic computed tomography alignment (with video),” J. Visc. Surg., vol. 156, pp. 261–262, 2019, doi: 10.1016/j.jviscsurg.2019.01.009.
A. Shvets, A. Rakhlin, A. Kalinin, and V. Iglovikov, “Automatic instrument segmentation in robot-assisted surgery using deep learning,” In 2018 17th IEEE international conference on machine learning and applications (ICMLA), 2018, pp. 624–628, doi: 10.1109/ICMLA.2018.00100.
T. A. lshirbaji, et al. “A convolutional neural network with a two-stage LSTM model for tool presence detection in laparoscopic videos,” Curr Dir Biomed Eng, 2020, doi: 10.1515/cdbme-2020-0002.
K. Jo, Y. Choi, J. Choi, JW. Chung. “Robust real-time detection of laparoscopic instruments in robot surgery using convolutional neural networks with motion vector prediction,” Appl Sci., vol. 9, p. 2865, 2019.
S. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio, “Generating sentences from a continuous space,” in SIGNLL, pp. 10-21, 2016.
P. Phutane, E. Buc, K. Poirot, E. Ozgur, D. Pezet, A. Bartoli, B. Le Roy, “Preliminary trial of augmented reality performed on a laparo-scopic left hepatectomy,” Surg. Endosc, vol. 32, pp. 514–515, 2018, doi: 10.1007/s00464-017-5733-4.
K. Cho, B. Merrienboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: Encoder–decoder approaches,” Syntax, Semantics and Structure in Statistical Translation, 2014.
P. Mascagni, C. Fiorillo, T. Urade, T. Emre, T. Yu, T. Wakabayashi, E. Felli, S. Perretta, L. Swanstrom, D. Mutter, et al. “Formalizing video documentation of the Critical View of Safety in laparoscopic cholecystectomy: A step towards artificial intelligence assistance to improve surgical safety,” Surg. Endosc., vol. 34, pp. 2709–2714, 2019, doi: 10.1007/s00464-019-07149-3.
P. Mascagni, A. Vardazaryan, D. Alapatt, T. Urade, T. Emre, C. Fiorillo, P. Pessaux, D. Mutter, J. Marescaux, G. Costamagna, et al. “Artificial Intelligence for Surgical Safety: Automatic Assessment of the Critical View of Safety in Laparoscopic Cholecystectomy Using Deep Learning,” Ann. Surg., vol. 275, pp. 955–961, 2022, doi: 10.1097/SLA.0000000000004351.
A. A. Shvets, A. Rakhlin, A. A. Kalinin, V. I Iglovikov, “Automatic Instrument Segmentation in Robot-Assisted Surgery using Deep Learning,” In Proceedings of the 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), 2018, pp. 624–628, doi: 10.1109/ICMLA.2018.00100.
T. Kurmann, P. Márquez-Neila, M. Allan, S. Wolf, R. Sznitman, “Mask then classify: Multi-instance segmentation for surgical instruments,” Int. J. Comput. Assist. Radiol. Surg., vol. 16, pp. 1227–1236, 2021, doi: 10.1007/s11548-021-02404-2.
O. Dergachyova, D. Bouget, A. Huaulme, X. Morandi, and P. Jannin, “Automatic data-driven real-time segmentation and recognition of surgical workflow,” IJCARS, vol. 11, no. 6, pp. 1081–1089, 2016.
D. A. Hashimoto, G. Rosman, and O. R. Meireles, Artificial Intelligence in Surgery: Understanding the Role of AI in Surgical Practice, 1st ed. McGrawHill Education / Medical, 2021.
W. Guo, X. Gu, and Q. Fang, “Comparison of performances of conventional and deep learning-based methods in segmentation of lung vessels and registration of chest radiographs,” Radiol Phys Technol, vol. 14, pp. 6–15, 2021.
T. Collins, D. Pizarro, S. Gasparini, N. Bourdel, P. Chauvet, M. Canis, L. Calvet, A. Bartoli, “Augmented Reality Guided Laparoscopic Surgery of the Uterus,” IEEE Trans. Med. Imaging, vol. 40, pp. 371–380, 2020, doi: 10.1109/TMI.2020.3027442.
E. Özgür, B. Koo, B. Le Roy, E. Buc, and A. Bartoli, “Preoperative liver registration for augmented monocular laparoscopy using backward–forward biomechanical simulation,” Int. J. Comput. Assist. Radiol. Surg., vol. 13, pp. 1629–1640, 2018, doi: 10.1007/s11548-018-1842-3.
T. Tokuyasu, Y. Iwashita, and Y. Matsunobu, “Development of an artificial intelligence system using deep learning to indicate anatomical landmarks during laparoscopic cholecystectomy,” Surg Endosc, vol. 35, pp. 1651–8, 2021.
D. Kitaguchi, N. Takeshita, and H. Matsuzaki, “Automated laparoscopic colorectal surgery workflow recognition using artificial intelligence: Experimental research,” Int J Surg, vol. 79, pp. 88–94, 2020.
C. Matava, E. Pankiv, and S. Raisbeck, “A Convolutional Neural Network for Real Time Classification, Identification, and Labelling of Vocal Cord and Tracheal Using Laryngoscopy and Bronchoscopy Video,” J Med Syst, vol. 44, no. 44, 2020.
A. P. Twinanda, S. Shehata, and D. Mutter, “EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos,” IEEE Trans Med Imaging, vol. 36, pp. 86–97, 2017.
S. M. Zadeh, T. Francois, and L. Calvet, “SurgAI: deep learning for computerized laparoscopic image understanding in gynaecology,” Surg Endosc, vol. 34, pp. 5377–83, 2020.
F. Yu, G. S. Croso, and T. S. Kim, “Assessment of Automated Identification of Phases in Videos of Cataract Surgery Using Machine Learning and Deep Learning Techniques,” JAMA Netw Open, vol. 2, p. e191860, 2019.
B. Abd-El-Atty, A. M. Iliyasu, H. Alaskar, and A. A. A. El-Latif, “A Robust Quasi-Quantum Walks-based Steganography Protocol for Secure Transmission of Images on Cloud-based E-healthcare Platforms,” Sensors, vol. 20, no. 11, p. 3108, May 2020, doi: 10.3390/s20113108.
M. Ahmed, S. Masood, M. Ahmad, and A. A. A. El-Latif, “Intelligent Driver Drowsiness Detection for Traffic Safety Based on Multi CNN Deep Model and Facial Subsampling,” IEEE Trans. Intell. Transp. Syst, pp. 1-10, 2021, doi: 10.1109/TITS.2021.3134222.
M. Higgins, C. Madan, and R. Patel, “Development and decay of procedural skills in surgery: A systematic review of the effectiveness of simulation-based medical education interventions,” The Surgeon, vol. 19, no. 4, 2021.
M. Hammad, “Deep Learning Models for Arrhythmia Detection in IoT Healthcare Applications,” Comput. Electr. Eng, vol. 100, p. 108011, 2022, doi: 10.1016/j.compeleceng.2022.108011.
K. Jo, Y. Choi, J Choi, and JW. Chung “Robust real-time detection of laparoscopic instruments in robot surgery using convolutional neural networks with motion vector prediction,” Appl Sci., vol. 9, p. 2865, 2019.
P. Chauvet, N. Bourdel, L. Calvet, B. Magnin, G. Teluob, Canis, M. A. Bartoli, “Augmented Reality with Diffusion Tensor Imaging and Tractography during Laparoscopic Myomectomies,” J. Minim. Invasive Gynecol., vol. 27, pp. 973–976, 2019, doi: 10.1016/j.jmig.2019.11.007.
X. Kong, Y. Jin, Q. Dou, Z. Wang, Z. Wang, B. Lu, E. Dong, Y.-H. Liu, D. Sun, “Accurate instance segmentation of surgical instruments in robotic surgery: Model refinement and cross-dataset evaluation,” Int. J. Comput. Assist. Radiol. Surg., vol. 16, pp. 1607–1614, 2021, doi: 10.1007/s11548-021-02438-6.
X. Huang, S. G. McGill, B. C. Williams, L. Fletcher, and G. Rosman, “Uncertainty-aware driver trajectory prediction at urban intersections,” in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 9718-9724, doi: 10.1109/ICRA.2019.8794282.
X. Huang et al., “HYPER: Learned hybrid trajectory prediction via factored inference and adaptive sampling,” in 2022 International Conference on Robotics and Automation (ICRA), pp. 2906-2912, 2022, doi: 10.1109/ICRA46639.2022.9812254.
D. Kitaguchi, N. Takeshita, H. Hasegawa, M. Ito, “Artificial intelligence-based computer vision in surgery: Recent advances and future perspectives,” Ann. Gastroenterol. Surg., vol. 6, pp. 29–36, 2021, doi: 10.1002/ags3.12513.
B. Ivanovic, K. Leung, E. Schmerling, and M. Pavone, “Multimodal deep generative models for trajectory prediction: A conditional variational autoencoder approach,” RA-L, vol. 6, no. 2, pp. 295–302, 2020.
A. Attanasio, B. Scaglioni, E. De Momi, P. Fiorini, and P. Valdastri, “Autonomy in Surgical Robotics,” Annu. Rev. Control. Robot. Auton. Syst., vol. 4, pp. 651–679, 2021, doi: 10.1146/annurev-control-062420-090543.
A. Kanakatte, A. Ramaswamy, J. Gubbi, A. Ghose, B. Purushothaman Surgical tool segmentation and localization using spatio-temporal deep network. In: 2020 42nd annual international conference of the IEEE engineering in medicine & biology society (EMBC), 2020, p. 1658–1661, doi: 10.1109/EMBC44109.2020.9176676.
S. Rashid, A. Ahmed, I. A. Barazanchi, and Z. A. Jaaz, “Clustering algorithms subjected to K-mean and gaussian mixture model on multidimensional data set,” Period. Eng. Nat. Sci, vol. 7, no. 2, pp. 448-457, 2019.
A. Jin, S. Yeung, J. Jopling, J. Krause, D. Azagury, A. Milstein, and L. Fei-Fei, “Tool detection and operative skill assessment in surgical videos using region-based convolutional Neural Networks,” In 2018 IEEE winter conference on applications of computer vision (WACV), 2018.
S. Rashid, A. Ahmed, I. A. Barazanchi, A. Mhana, and H. Rasheed, “Lung cancer classification using data mining and supervised learning algorithms on multi-dimensional data set,” Period. Eng. Nat. Sci, vol. 7, no. 2, pp. 438-447, 2019.
M. Volkov, DA. Hashimoto, G. Rosman, et al., “Machine learning and coresets for automated realtime video segmentation of laparoscopic and robot-assisted surgery,” IEEE International Conference on Robotics and Automation (ICRA), 2017 pp. 754–759.
A. A. Gumbs, I. Frigerio, G. Spolverato, R. Croner, A. Illanes, E. Chouillard, and E. Elyan, “Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?,” Sensors, vol. 21, p. 5526, 2021, doi: 10.3390/s21165526.
M. F. S. Sabir, “An Automated Real-Time Face Mask Detection System Using Transfer Learning with Faster-RCNN in the Era of the COVID-19 Pandemic,” Comput. Mater. Contin, vol. 71, no. 2, pp. 4151-4166, 2022, doi: 10.32604/cmc.2022.017865.
Y. K. Salih, O. H. See, S. Yussof, A. Iqbal, and S. Q. M. Salih, “A proactive fuzzy-guided link labeling algorithm based on MIH framework in heterogeneous wireless networks,” Wirel. Pers. Commun, vol. 75, no. 4, pp. 2495-2511, 2014, doi: 10.1007/s11277-013-1479-z.
H. Tao et al., “A Newly Developed Integrative Bio-Inspired Artificial Intelligence Model for Wind Speed Prediction,” IEEE Access, vol. 8, pp. 83347-83358, 2020, doi: 10.1109/ACCESS.2020.2990439.
G. Wang and S. Wang, “Surgical tools detection based on training sample adaptation in laparoscopic videos,” IEEE Access, vol. 8, pp. 181723–181732, 2020, doi: 10.1109/ACCESS.2020.3028910.
L. Tawalbeh, F. Muheidat, M. Tawalbeh, M. Quwaider, and A. A. A. El-Latif, “Edge enabled IoT system model for secure healthcare,” Measurement, vol. 191, p. 110792, Mar. 2022, doi: 10.1016/j.measurement.2022.110792.
B Namazi, G. Sankaranarayanan, and V. Devarajan, “LapTool-Net: a contextual detector of surgical tools in laparoscopic videos based on recurrent convolutional neural networks,” arXiv preprint, arXiv:1905.08983, 2019.
P. Shi, Z. Zhao, S. Hu, and F. Chang “Real-time surgical tool detection in minimally invasive surgery based on attention-guided convolutional neural network,” IEEE Access, vol. 8, pp. 228853–228862, 2020, doi: 10.1109/ACCESS.2020.3046258.
F. Abbas, M. Yasmin, M. Fayyaz, M. A. Elaziz, S. Lu, and A. A. A. El-Latif, “Gender Classification Using Proposed CNN-Based Model and Ant Colony Optimization,” Mathematics, vol. 9, no. 19, p. 2499, Oct. 2021, doi: 10.3390/math9192499.
M.-H. Laves, J. Bicker, L. A. Kahrs, and T. Ortmaier, “A dataset of laryngeal endoscopic images with comparative study on convolution neural network-based semantic segmentation,” Int. J. Comput. Assist. Radiol. Surg., vol. 14, 483–492, 2019, doi: 10.1007/s11548-018-01910-0.
P. Mota, N. Carvalho, E. Carvalho-Dias, M. Joao Costa, J. Correia- Pinto, and E. Lima, “Video-based surgical learning: improving trainee education and preparation for surgery,” J Surg Educ., vol. 75, no. 3, pp. 828–835, 2018, doi: 10.1016/j.jsurg.2017. 09.027.
L. Li, H. Huang, and X. Jin, “AE-CNN classifcation of pulmonary tuberculosis based on CT images,” In 2018 9th international conference on information technology in medicine and education (ITME), 2018, doi: 10.1109/itme.2018. 00020.
Z. Xiao, R. Huang, Y. Ding, T. Lan, R. Dong, Z. Qin, X. Zhang, and W. Wang “A deep learning-based segmentation method for brain tumor in MR images,” In 2016 IEEE 6th international conference on computational advances in bio and medical sciences (ICCABS), 2016, doi: 10.1109/iccabs.2016.7802771.
S. Wang, A. Raju, and J. Huang, “Deep learning based multi-label classification for surgical tool presence detection in laparoscopic videos,” In 2017 IEEE 14th international symposium on biomedical imaging (ISBI 2017), 2017. p. 620–623, doi: 10.1109/ISBI.2017.7950597.
I. A. Elgendy, W.-Z. Zhang, H. He, B. B. Gupta, and A. A. A. El-Latif, “Joint computation offloading and task caching for multi-user and multi-task MEC systems: reinforcement learning-based algorithms,” Wirel. Networks, vol. 27, no. 3, pp. 2023-2038, Apr. 2021, doi: 10.1007/s11276-021-02554-w.
J. Nour, A. Tamer, and M. Knut. Predicting surgical phases using CNN-NARX neural network,” Curr Dir Biomed Eng., vol. 5, pp. 405–407, 2019, doi: 10.1515/cdbme-2019-0102.
B. Zhang, S. Wang, L. Dong, and P. Chen, “Surgical tools detection based on modulated anchoring network in laparoscopic videos,” IEEE Access, vol. 8, pp. 23748–23758, 2020, doi: 10.1109/ACCESS.2020.2969885.
A. A. Shvets, A. Rakhlin, A. A. Kalinin, and V. I. Iglovikov, “Automatic instrument segmentation in robot-assisted surgery using deep learning,” In 2018 17th IEEE international conference on machine learning and applications (ICMLA), 2018, p. 624–628, doi: 10.1109/ICMLA.2018.00100.
D. Kitaguchi, N. Takeshita, H. Matsuzaki, H.Takano, Y. Owada, T. Enomoto, T. Oda, H. Miura, T. Yamanashi, M. Watanabe, et al. “Real-time automatic surgical phase recognition in laparoscopic sigmoidectomy using the convolutional neural network-based deep learning approach,” Surg. Endosc., vol. 34, pp. 4924–4931, 2019, doi: 10.1007/s00464-019-07281-0.
D. Sarikaya, J. J. Corso, and K. A. Guru, “Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection,” IEEE Trans. Med. Imaging, vol. 36, pp. 1542–1549, 2017.
T. François, L. Calvet, S. M. Zadeh, D. Saboul, S. Gasparini, P. Samarakoon, N. Bourdel, and A. Bartoli, “Detecting the occluding contours of the uterus to automatise augmented laparoscopy: Score, loss, dataset, evaluation and user study,” Int. J. Comput. Assist. Radiol. Surg., vol. 15, pp. 1177–1186, 2020, doi: 10.1007/s11548-020-02151-w.
S. M. Zadeh, T. Francois, L. Calvet, P. Chauvet, M. Canis, A. Bartoli, and N. Bourdel, “SurgAI: Deep learning for computerized laparoscopic image understanding in gynaecology,” Surg. Endosc., vol. 34, pp. 5377–5383, 2020, doi: 10.1007/s00464-019-07330-8.
DOI: https://doi.org/10.18196/jrc.v3i6.16201
Refbacks
- There are currently no refbacks.
Copyright (c) 2023 Saadya Fahad Jabbar
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Journal of Robotics and Control (JRC)
P-ISSN: 2715-5056 || E-ISSN: 2715-5072
Organized by Peneliti Teknologi Teknik Indonesia
Published by Universitas Muhammadiyah Yogyakarta in collaboration with Peneliti Teknologi Teknik Indonesia, Indonesia and the Department of Electrical Engineering
Website: http://journal.umy.ac.id/index.php/jrc
Email: jrcofumy@gmail.com