An objective benchmark is ready for prime time.
The use of mannequin simulators for the training of medical personnel is not a new concept. The Harvey cardiology mannequin was first demonstrated in 1968 and has been widely used to train medical students and residents. Subsequently, simulators became more realistic (SimMan [Laerdal, Wappingers Falls, NY] for example), but did not achieve widespread acceptance. More recently, specialty trainers for endoscopy, laparoscopy, arthroscopy, and endovascular interventions have been developed. Many of these simulators employ realistic imaging, tactile haptics, full case management, and the ability to capture metrics of performance. Such sophisticated simulators are now being used not only to train medical students, but also to train experienced operators in new procedures and to test complication management skills.
Several potential applications are apparent for the current generation of procedure simulators:
- Training and objective testing of students, residents, and fellows
- Training of experienced physicians in new procedures
- Use in certification examinations (eg, ABIM certification)
- Use in hospital privileging
- Use in identifying competent physicians for insurance panels
- Use in remedial training for physicians with quality issues
A growing pool of evidence suggests that these applications are valid. In a randomized trial of laparoscopic cholecystectomy, 16 surgeons were observed on two surgeries. Between these two surgeries, half received intensive simulator training and half did not. Those with simulator training were significantly faster, had fewer unnecessary movements, and had fewer errors (P=.003).1 A similar study of laparoscopic cholecystectomy in 16 surgical residents revealed that those randomized to receive simulator training were 29% faster and six times less likely to make errors (P=.008).2 A third study randomizing residents to simulation for laparoscopic surgery documented faster performance, fewer errors, and higher accuracy.3 In a study of flexible bronchoscopy, fellows who received 4 hours of group instruction and 4 hours of individual simulation training performed at a level equivalent to that of physicians with experience of 2 years and more than 200 procedures.4
The ability of simulation to differentiate levels of experience has also been tested. In a study of 75 surgical attendings, residents, and fellows, there was clear separation in performance of colonoscopy procedures. Using multiple metrics including efficiency, rate of polypectomy, and degree of colon visualized, the simulation could effectively segment skill levels.5 Similar findings were observed for ureteral stone removal. Novices and low- and high-volume operators could be distinguished. Furthermore, those who received training had better performance on subsequent patient procedures.6
A review by Gallagher and Cates acknowledges the importance of obtaining metrics that separate novices from experts. They note that "An optimum approach to using simulation for training would be to first establish an objective benchmark on the simulator, based on the performance of experienced operators."7 The Society for Cardiovascular Angiography and Interventions (SCAI) agrees and is developing a program for percutaneous coronary intervention.
THE SCAI EXPERIENCE
The SCAI was founded in 1978 with the mission of promoting excellence in invasive and interventional cardiovascular medicine through physician education and the enhancement of quality standards to enhance patient care. When endovascular simulators were developed, the SCAI recognized their potential importance to the field, which is very technically demanding and in which complications can be life-threatening. The SCAI believed that setting a national benchmark using objective metrics would be critical to widespread use of simulators for more than just novice training. In 2004, the SCAI partnered with Medical Simulation Corporation (Denver, CO) to develop a series of completely simulated patient cases that could form the basis for establishing a national benchmark. Each case presents history, physical, and laboratory data (Figure 1), and requires procedure planning, procedure execution, complication management (many complications are "forced"), and final patient disposition. During the simulation, the operator is in a virtual catheterization lab complete with sounds, monitors, and endovascular catheters (Figure 2). Furthermore, as in real life, the operators must manage the patient's medications, fluoroscopy, laboratory monitoring, and queries from the nurses. All operator actions are tracked and recorded.
Once these cases were developed and tested, a panel of 23 experts nominated by SCAI members and representing all varieties of practice situations were tested. Their results were tabulated and, from their data, national benchmark standards were established. Subsequent physicians performing these simulations can then have their performance compared to that of the experts, providing an objective and quantifiable measure of an individual physician's technical performance, complication management ability, and efficiency. A detailed report is provided to the physician, with suggestions for areas of improvement (Figure 3).
How can this tool help you? Many of us involved in simulation envision physicians performing simulation exercises on a regular basis, perhaps yearly, to keep their knowledge of complication management current, and to provide ongoing feedback for self-education. In more controversial situations, individual physicians may need to "validate" their capabilities using an objective tool devoid of local politics. Perhaps a hospital is concerned that the quality in their lab is slipping. By having objective data on the performance of their physicians, they could better focus their corrective efforts. With pay for performance in the wings, perhaps an objective measure of quality performance of endovascular procedures could be incorporated into an overall quality improvement program. Several national credentialing bodies are investigating the use of simulation for those specialties with heavy emphasis on procedural competence. This is expected to become a reality in the near future.
We believe this benchmark project provides invasive and interventional physicians, their hospital, and ultimately their patients with a practical tool to be used as part of their overall quality assurance program. Patients deserve physicians who are as well trained and as prepared as they can possibly be. We believe simulation is ready for prime time.
John McB. Hodgson, MD, FSCAI, is Professor of Medicine and Chief of Academic Cardiology, St. Joseph's Hospital and Medical Center, Phoenix, Arizona. He has disclosed that he receives research funding from the St. Joseph's Foundation. Dr. Hodgson may be reached at (602) 406-3915; email@example.com.
This project was supported in part by a grant from the St. Joseph's Foundation, Phoenix, Arizona