Home

Research

People

Publication

Education

News

Contact

Resources

 
Video Understanding Methods for Surgical Tool Detection and Tracking Dataset (Public Access)

 

This dataset is a result of work done at Automation, Robotics and Mechatronics Laboratory (ARMLAB) and Computer Integrated Surgery Research Laboratory (CIS-LAB) , State University of New York (SUNY) at Buffalo. It has videos of real surgical sequences performed using Intuitive Surgical Inc.'s Da-Vinci robot with manual annotations for tools and their attributes (open/closed and Blood Stained/ Not Stained). This data and associated code is released under GNU General Public License, latest version of which can be accessed at http://www.gnu.org/licenses/gpl.html.

Contact: Suren Kumar (surenkum@buffalo.edu/ ARMLAB Admin (armlabwebmaster@gmail.com)

Dataset Description: This dataset has 12 small videos of surgical activity as captured by endoscopic camera in sub directory "videos". Sub directory "models" has DPM [1] models for visual detection of tools from images. Attributes folder under annotations sub folder has annotated tool bounding boxes and attributes details. Code folder has helpful codes for visualization and getting detections. Video1.mpg in the dataset does not have any associated attributes.

Annotation Description: An annotation file is available for each attribute of each tool in each video. So, for a video that typically contains two tools, 4 associated annotation files are available in “annotations” directory that correspond to two tools in each video and two files for attributes (two attributes for each tool are considered presently) corresponding to each tool.


(a) Tool open/close attribute is in files-- videofile_toolPosition_TXXXX.csv, where videofile is the name of video file. toolPosition is "L" or "R" depending on which side (left/right) of the video does tool appear first.
(b) Annotation files corresponding to blood stained attributes has _Blood appended to the filename before .csv filename. 

The column format of each .csv file in annotation folder is organized into: (x, y, w, h, Attribute, Marked) where,

  • (x, y) is the top left corner of the tool in the image,

  • w is the width and h is height in image frame for a tool annotated by a bounding box,

  • Attribute is 1 for Open, 0 for Closed and 1 for blood stained, 0 for not blood stained.


Code: Additionally, we have provided useful MATLAB codes for visualization of tool attributes and detection that are placed in \code\utilities.
(a) Attribute visualization (attribute_visualize.m): Run this script with default video name for visualization of attributes. To run it for a different video, one only needs to change the variable “videofilename” to the corresponding name of video file.
A frame that contains a sample manual annotation (for the tool bounding boxes) from this dataset is given in the graphic below. Using this function, will enable to generate such a plot for a sequence of frames (of a surgical video) alongwith the bounding boxes and the corresponding attribute values.


(b) Tool detection (detection_test.m): This script contains code for implementing our tool detection algorithm, adopted from [1], for a test image. This code has been tested on Windows 7+Matlab 2013+Visual Studio 2010 (as compiler for mex) and Linux+Matlab 2009+g++ (as compiler for mex)
 

If you would like to download a copy of the dataset, please fill the form below:

 


If you find this dataset helpful in your research, please cite our papers [2] and [3]. For any suggestions, corrections, please contact us at the contact email listed on top of this page. If you would like to receive information on future updates, we also recommend you provide your contact details separately to the email IDs listed above with the subject line: "subscribe to open surgical dataset updates".

References
[1] P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan. Object Detection with Discriminatively Trained Part Based Models, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, No. 9, September 2010, Code: http://cs.brown.edu/~pff/latent-release4.
[2] Video-based Framework for Safer and Smarter Computer Aided Surgery, S Kumar, M Sathia Narayanan, S Misra, S Garimella, P Singhal, J Corso, V Krovi, The Hamlyn Symposium on Medical Robotics, 107-108, 2013.
[3] Product of Tracking Experts for Visual Tracking of Surgical Tools. S Kumar, M Sathia Narayanan, P Singhal, J Corso, V Krovi, 9th IEEE International Conference on Automation Science and Engineering (CASE), 2013.

 

 
Students Involved:

- Suren Kumar, PhD Candidate, University at Buffalo

- Seung-kook Jun, PhD Candidate, University at Buffalo

- Madususdanan Sathia Narayanan, PhD Candidate, University at Buffalo

- Priyanshu Agarwal, MS, University at Buffalo [Graduated]

- Sukumar Misra, Surgical Intern [Compeleted]

 

 Movies :

Time Study based Skill Assessment

- Motion study for surgical skill assessment comprises of different steps including motion segmentation, discrete Therblig definition, motion analysis and automated classification/ recognition schemes.

- File Size: 22.3MB [Download]

Surgical Tool Visual Tracking Framework

- This video illustrates our recent work on video-based surgical tool detection and tracking for a real robotic hysterectomy surgical sequence.

- File Size: 22.3MB [Download]

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 Related Publications - Journals:

[01]

Jun, S.-K., Narayanan, M.S., Garimella, S., Singhal, P., and Krovi, V., "Evaluation of Robotic Minimally Invasive Surgical Skills using Motion Studies", Springer Journal of Robotic Surgery, pp. 1-9, 2013/07/14 2013.[BIB | RIS]

[PDF]

 

 Related Publications - Conference Proceedings:

[01]

Kumar, S., Narayanan, M.S., Singhal, P., Corso, J., and Krovi, V., “Product of Tracking Experts for Surgical Tool Visual Tracking,” 2013 IEEE Conference on Automation Science and Engineering, August 17-21 2013, Wisconsin, MA.

[PDF]

[02]

Kumar, S., Narayanan, M.S., Garimella, S. MD, Singhal, P. MD,  Corso, J., and Krovi, V.,  “Novel Computer Aided Surgical Workflow Using Video–Based Detection And Assessment Methods,” ASME/FDA 2013 1st Annual Frontiers in Medical Devices, September 11-13, 2013, Washington, DC, USA.

[PDF]

[03]

Kumar, S., Narayanan, M.S., Misra, S., Garimella, S., Singhal, P. MD,  Corso, J., and Krovi, V., "Video-based Framework for Safer and Smarter Computer Aided Surgery,” 2013 Hamlyn Symposium on Medical Robotics, London UK, 22-25 Jun, 2013.

[PDF]

[04]

Kumar, S., Narayanan, M.S., Misra, S., Garimella, S., Singhal, P., Corso, J.,  and Krovi, V., "Vision based Decision-Support and Safety Systems for Robotic Surgery", 2013 Medical Cyber Physical Systems Workshop, Philadelphia, PA, April 8, 2013.

[PDF]

[05]

Jun, S.-K., Narayanan, M.S., Eddib, A., MD, Garimella, S., MD, Singhal, P, MD, and Krovi, V., “Robotic Minimally Invasive Surgical Skill Assessment based on Automated Video-Analysis Motion Studies”, 2012 IEEE International Conference on Biomedical Robotics and Biomechatronics, Roma, Italy, Jun 24-28, 2012. [BIB | RIS]

[PDF]

[06]

Jun, S.-K., Narayanan, M.S., Eddib, A., MD, Garimella, S., MD, Singhal, P, MD, and Krovi, V., “Minimally Invasive Surgical Skill Assessment by Video-Motion Analysis”, 2012 5th Hamlyn Symposium on Medical Robotics, 2012, London, UK, Jun 30-Jul 2. [BIB | RIS]

[PDF]

[07]

Jun, S.-K., Narayanan, M.S., Eddib, A., MD, Garimella, S., MD, Singhal, P, MD, and Krovi, V., “Evaluation of Robotic Minimally Invasive Surgical Skills using Motion Studies”, 2012 Performance Metrics for Intelligent Systems (PerMIS'12) Workshop, March 20-22, 2012, College Park, MD. [BIB | RIS]

[PDF]

Questions or comments regarding the website, please contact the webmaster.

Last Updated: September 25, 2013