Quantitative Skill Assessment of Robotic Minimally Invasive Surgical Skills (rMIS) using Motion Studies

Surgical proficiency engenders merger of sensory and cognitive capabilities and conducting systematic assessment in this context has always been a topic of considerable importance.
Robotic minimally-invasive-surgery (rMIS) is the fastest growing segment of computer-aided surgical systems today and has often been heralded as the new revolution in healthcare industry. However, the surgical performance-evaluation paradigms have always failed to keep pace with the advances of surgical technology. The biggest challenges to assessment and accreditation of surgeons include (i) creating appropriately rich and diverse clinical settings (real or virtual); as well as (ii) developing uniform, repeatable, stable, verifiable performance metrics; both at manageable financial levels for ever  increasing cohorts of trainees.

The surgical training programs are considered to be inconsistent, not-proven and non-uniform and normally includes operating on surrogate phantoms ranging from cadavers to real-life surgeries, animal models to plastic mannequins to most recently simulated/virtual environments. Such virtual reality trainers leverage an apprenticeship model and entails subjective or at best semi-objective evaluation of surgical performance by an expert surgeon. Over the past decade, the ACGME (Accreditation Council for Graduate Medical Education) has espoused development of a cost-efficient proficiency-based curriculum, with an emphasis on simulation methodologies and quantitative skills-assessment tools, to bypass the limitations in the current apprenticeship-based system. In addition, the growth of computer integration and data acquisition in minimally-invasive-surgery (MIS) especially in the form of rMIS offers a unique set of opportunities to comprehensively address this situation.


In this work, we examine the extension of traditional manipulative skill assessment with deep roots in performance evaluation in manufacturing industries for applicability to robotic surgical skill evaluation. The traditional time and motion studies are based on the hypothesis that: any manipulation or assembly task can be subdivided into smaller individual units called “Therbligs”. These “Therbligs” allow for decomposition of a complicated manual task into sub-parts that could then be individually examined. This decomposition potentially allows for a finite state automaton representation of a complex activity that could form the discrete basis for linguistic representation as well as fault-detection and correction. Intra- and inter-user variance on various performance metrics can by analyzed by studying surgeons’ performance over each sub-tasks. Additional metrics on tool-motion measurements, motion economy, and handed-symmetry can be similarly expanded over this temporal segmentation to help characterize performance. Our studies analyzed video recordings of surgical task performance in two settings:  First, we analyzed video data for two representative manipulation exercises (peg board and pick-and-place) on a da Vinci surgical (SKILLS) simulator to afford a relatively-controlled and standardized testbed for surgeons with varied experience-levels. Second task-sequences from real surgical videos were analyzed with a list of predefined “Therbligs” in order to investigate its usefulness for practical implementation.

An automated surgical expertise evaluation method based on these well-established motion studies methodologies, especially for MIS procedures. This method relies on segmenting of a primary task into sub-tasks, which can subsequently be analyzed by statistical analyses of micro-motions. We conducted motion studies using: (A) manual annotation process by experts (to serve as a benchmark); and (B) automated kinematic-analysis-of-video techniques; for motion economy, repeatability as well as dexterity. The da Vinci SKILLS simulator was used for our studies to serve as a uniform and standardized testbed. Surgeons with varied levels of expertise were recruited to perform two representative simplified tasks (Peg Board and Pick & Place). The automated kinematic analysis of video was compared with the ground truth data (obtained by manual labeling) using misclassification rate and true classification confusion matrix. Future studies aimed towards analyzing real surgical procedures and extending the existing framework using probabilistic approaches are already underway.


 Students Involved:

- Seung-kook Jun, PhD Candidate, University at Buffalo

- Madususdanan Sathia Narayanan, PhD Candidate, University at Buffalo

- Priyanshu Agarwal, MS, University at Buffalo [Graduated]


 Movies :

Time Study based Skill Assessment

- Motion study for surgical skill assessment comprises of different steps including motion segmentation, discrete Therblig definition, motion analysis and automated classification/ recognition schemes.

- File Size: 22.3MB [Download]








 Related Publications - Conference Proceedings:


Jun, S.-K., Narayanan, M.S., Eddib, A., MD, Garimella, S., MD, Singhal, P, MD, and Krovi, V., “Robotic Minimally Invasive Surgical Skill Assessment based on Automated Video-Analysis Motion Studies”, 2012 IEEE International Conference on Biomedical Robotics and Biomechatronics, Roma, Italy, Jun 24-28, 2012. [BIB | RIS]



Jun, S.-K., Narayanan, M.S., Eddib, A., MD, Garimella, S., MD, Singhal, P, MD, and Krovi, V., “Evaluation of Robotic Minimally Invasive Surgical Skills using Motion Studies”, 2012 Performance Metrics for Intelligent Systems (PerMIS'12) Workshop, March 20-22, 2012, College Park, MD. [BIB | RIS]


Questions or comments regarding the website, please contact the webmaster.

Last Updated: April 21, 2012