|Year : 2020 | Volume
| Issue : 4 | Page : 179-187
|Development and validation of metric-based-training to proficiency simulation curriculum for upper gastrointestinal endoscopy using a novel assessment checklist
Nahla Azzam1, Nehal Khamis2, Majid Almadi1, Faisal Batwa3, Fahad Alsohaibani4, Abdulrahman Aljebreen1, Ahmad Alharbi5, Yasser Alaska6, Turki Alameel7, Peter Irving8, Richard M Satava9
1 Department of Medicine, Division of Gastroenterology, King Saud University Medical City, King Saud University, Riyadh, Kingdom of Saudi Arabia
2 Saudi Commission for Health Specialties; King Saud University Clinical Skills and Simulation Center, Riyadh, Kingdom of Saudi Arabia; Departments of Pathology and Medical Education, College of Medicine, Suez Canal University, Egypt
3 Department of Medicine, Division of Gastroenterology, King Saud bin AbdulAziz University for Health Sciences, Jeddah, Kingdom of Saudi Arabia
4 Department of Medicine, King Faisal Specialist Hospital and Research Center, Riyadh, Kingdom of Saudi Arabia
5 Department of Medicine, King Faisal Specialist Hospital and Research Center, Jeddah, Kingdom of Saudi Arabia
6 King Saud University Clinical Skills and Simulation Center; Department of Emergency Medicine, King Saud University, Riyadh, Kingdom of Saudi Arabia
7 Department of Medicine, King Fahad Specialist Hospital, Dammam, Kingdom of Saudi Arabia
8 London Digestive Health, London Hospital Medical College, London, United Kingdom
9 Department of Surgery, University of Washington Medical Center, Seattle, Washington, United States of America
Click here for correspondence address and email
|Date of Submission||19-Mar-2020|
|Date of Acceptance||05-Jun-2020|
|Date of Web Publication||21-Jul-2020|
| Abstract|| |
Background/Aims: This study aimed to design a structured simulation training curriculum for upper endoscopy and validate a new assessment checklist.
Materials and Methods: A proficiency-based progression stepwise curriculum was developed consisting of didactic, technical and non-technical components using a virtual reality simulator (VRS). It focused on: scope navigation, anatomical landmarks identification, mucosal inspection, retro-flexion, pathology identification, and targeting biopsy. A total of 5 experienced and 10 novice endoscopists were recruited. All participants performed each of the selected modules twice, and mean and median performance were compared between the two groups. Novices pre-set level of proficiency was set as 2 standard deviations below the mean of experts. Performance was assessed using multiple-choice questions for knowledge, while validated simulator parameters incorporated into a novel checklist; Simulation Endoscopic Skill Assessment Score (SESAS) were used for technical skills.
Results: The following VRS outcome measures have shown expert vs novice baseline discriminative ability: total procedure time, number of attempts for esophageal intubation and time in red-out. All novice trainees achieved the preset level of proficiency by the end of training. There were no statistically significant differences between experts' and trainees' rate of complications, landmarks identification and patient discomfort. SESAS checklist showed high degree of agreement with the VRS metrices (kappa = 0.83) and the previously validated direct observation of procedural skills tool (kappa = 0.90).
Conclusion: The Fundamentals of Gastrointestinal Endoscopy simulation training curriculum and its SESAS global assessment tool have been primarily validated and can serve as a valuable addition to the gastroenterology fellowship programs. Follow up study of trainee performance in workplaces is recommended for consequences validation.
Keywords: Assessment tool, colonoscopy, curriculum, endoscopy skills, gastroscopy, gastroenterology training, metrics, simulation, training to proficiency
|How to cite this article:|
Azzam N, Khamis N, Almadi M, Batwa F, Alsohaibani F, Aljebreen A, Alharbi A, Alaska Y, Alameel T, Irving P, Satava RM. Development and validation of metric-based-training to proficiency simulation curriculum for upper gastrointestinal endoscopy using a novel assessment checklist. Saudi J Gastroenterol 2020;26:179-87
|How to cite this URL:|
Azzam N, Khamis N, Almadi M, Batwa F, Alsohaibani F, Aljebreen A, Alharbi A, Alaska Y, Alameel T, Irving P, Satava RM. Development and validation of metric-based-training to proficiency simulation curriculum for upper gastrointestinal endoscopy using a novel assessment checklist. Saudi J Gastroenterol [serial online] 2020 [cited 2020 Sep 29];26:179-87. Available from: http://www.saudijgastro.com/text.asp?2020/26/4/179/290341
See accompanying Editorial
| Introduction|| |
The traditional apprenticeship model of education in endoscopy has relied mainly on learning through observation and performing under supervision. This approach of training has depended for many years on the duration of training and volume of procedures (threshold numbers) as metrics/surrogates for accomplishment of clinical expertise.
Simulation-based training provides the opportunity of learning in a safe, low risk environment and as such allows the trainees to learn from their own mistakes., Simulation in endoscopy has shown reasonable accepted face validity, however evidences for construct validity were reported in many studies as good performance measures related mainly to procedural time.
Evidences exist from multiple systematic reviews that simulation-based endoscopy (SBE) training (e.g., virtual reality (VR)) can be educationally effective in preparation of novices in diagnostic esophagogastroduodenoscopy, colonoscopy and/or sigmoidoscopy before the conventional patient-based endoscopy training.,,,, Results show that SBE training accelerates the learning curve of the novice trainees, improves performance and it has proved to be transferable to clinical practice.,,
However, King et al. concluded in their systematic review that “the optimal SBE training program has yet to be developed”, a conclusion that agrees with that of a Cochrane review  as well.
Many efforts reported from different international scientific societies have focused on either the development of the cognitive component of the training curriculum (e.g., Fundamentals of Laparoscopic Surgery (FLS) or the development of evaluation tools for the assessment of gastrointestinal (GI) endoscopy technical and non-technical competencies (e.g., American Society for Gastrointestinal Endoscopy (ASGE)).
Up to date, we are aware of only few standardized validated curricula for SBE endoscopic training. In response to the need for a standardized approach for GI endoscopy simulation training in the Saudi gastroenterology fellowship programs, a task force of subject-matter experts was formed to develop a comprehensive simulation-based curriculum for the “fundamentals of GI endoscopy” (FGE). This newly developed curriculum was piloted as a mandatory fundamental GI course conducted a week ahead of the start of the national gastroenterology fellowship program and prior to any patient-based GI endoscopy encounter. The aims of this manuscript are to: 1. Describe the developed curriculum in terms of competencies, general and specific objectives, educational and learner assessment methods; 2. Provide construct validation evidence of the developed curriculum tasks, with identification of which VR simulator (VRS) metrics are capable of discriminating between novices and experts and; 3. Validate the new innovative assessment endoscopic skill checklist developed by the task force group.
| Materials and Methods|| |
Phase I: Development of the course curriculum
A task force team was formed in December 2016, which consisted of national experts who were involved in previous pilot GI simulation courses (MA, NA, FB, AH), experienced endoscopists (FS, TT, AJ,), and simulation-based education experts (NK, YA, RS). A comprehensive metric-based-training-to-proficiency curriculum was designed by the working group using Khamis et al. (2016) model. The model and its worksheet were developed by integrating the criteria of educational effectiveness of simulation into the steps of curriculum development. It addresses the three competency domains of knowledge, technical and non-technical skills.
The main aim of the FGE course is to introduce the learners to the fundamental principles of upper GI endoscopy. The competencies, general objectives, outcome measures, and common errors that should be avoided are shown in [Table 2] and [Table 3]. These were developed through serial meetings by the task force team. The team has used a modified Delphi consensus process  in the duration between December 2016 till January 2018 to establish the content validity of the developed course. The Delphi panel included a group of 11 multinational experts; eight endoscopists (MM, NA, FB, AH, FS, TA, AJ, PI) and three simulation education experts (NK, YA, RS). To assure content validation, task deconstruction and review of the literature were used to develop the initial course content in a face-to-face meeting. Then two rounds of email voting were used to reach final consensus.
|Table 1: Upper GI endoscopy 'Simulation Endoscopic Skill Assessment Score' (SESAS) Checklist|
Click here to view
|Table 2: Competencies, General objectives, and Outcome Measures of the Developed FGE Curriculum|
Click here to view
|Table 3: List of the Common Errors for Gastrointestinal Endoscopy Reached by a Consensus of the Subject Matter Experts Designing the Course|
Click here to view
The outcome measures and content of the didactic part of the cognitive module of the curriculum were adapted from the validated Fundamentals of Endoscopic Surgery (FES) didactic online course (http://www.fesprogram.org/fes-didactic/).
The AccuTouch® Endoscopy Simulator (Immersion Corp., CA, and USA) was used by the task force team for the administration of the technical skill part of the fundamentals of endoscopy training.
The cognitive module was delivered through a series of interactive lectures, videos, and workshops that were developed and validated by the curriculum development team. This module introduces the trainees to the world of endoscopy, starting with lectures addressing the indications/contraindications of endoscopy, consent taking, levels of sedation, and how to stratify patients prior to sedation administration. These lectures were followed by a video on disinfection of the scopes then an interactive workshop on scope “parts, tools, and accessories”. This module prepared the trainees to be able to plan the endoscopy procedure in a stepwise fashion that mimics real life settings.
Phase II: Construct validation of the modules and setting proficiency benchmarks
A total of 15 subjects (5 experts and 10 post-graduate year-4 internal medicine residents; PGY4) were recruited for the study. An experienced endoscopist was defined as having performed more than 500 upper GI endoscopies and 200 colonoscopies. The 10 novice trainees (PGY4 residents) represented a subset of the residents who were accepted for a GI fellowship program in Saudi Arabia for the year 2018 and reported no previous endoscopy performance or completion of endoscopy training/rotation. Participants were also excluded if they previously attended upper endoscopy SBE training.
In December 2017, each of the 5 expert endoscopists completed two rounds on the simulator performing an upper GI endoscopy for a case of dyspepsia. The results of the simulator-generated 12 outcome measures and metrics of the procedure were recorded. These 12 outcome measures are: (1) Time: total procedure time, time to upper esophageal sphincter intubation, time to pylorus intubation, (2) Identification of anatomical landmarks: gastro-esophageal-junction, gastric fundus, angularis-incisura, major papilla and 3rd part of the duodenum, (3) Esophageal intubation: esophagus intubated with/without patient swallowing and number of attempts to intubate the esophagus, (4) Total amount of air inflated: total amount of air and air left, (5) Others: minimum depth of scope insertion, time in red-out and maximum force exerted by endoscope, (6) Pathology identification: visualization, location and lesion type, (7) Patient discomfort: the percentage of patient non-discomfort throughout the procedure, amount of time of mild and moderate discomfort, and (8) Complications: perforation, tracheal intubation and laryngeal edema. The level of proficiency required to pass each of the tasks of the VR simulator exercise was determined as two standard deviations (SDs) below the mean of the scores of the experts in two consecutive repetitions of the tasks.
From 20th to 25th of February 2018, the 10 PGY4 residents with no previous experience in endoscopy participated in the mandatory fundamentals of endoscopy course. As described before, they all received an orientation of the VR simulator followed by completion of the cognitive module. All trainees were required to pass a post-cognitive module multiple choice question (MCQ) test to determine eligibility to start the VR simulation technical (psychomotor) and non-technical skills training module. Following this they participated in a hands-on workshop for identification of scope parts, tools and accessories with demonstration of the steps of performing the upper GI endoscopy as shown in [Figure 1]. The technical module of the simulation course was facilitated by three expert endoscopists. Training on the communication and ethical behavior with the patient (non-technical skills components) was performed before the virtual simulation tasks, where the course facilitator played the role of simulated patient.
Trainees were allowed to practice with direct supervision and coaching. Avoidance of the common and critical errors (e.g. esophagus perforation, tracheal edema) was re-emphasized.
Deliberate repetitive practice was then permitted for all trainees where they repeated each of the two VR simulator tasks (basic upper endoscopy and a diagnostic biopsy) with formative feedback. Each of the trainees had to achieve technical proficiency in two consecutive trials of each task before proceeding to the next module/task.
Phase III: Assessment
Assessment of the cognitive module
For the cognitive skills, 10 MCQs were answered by the experts once before the course. The same 10-item MCQ test was then administered to the trainees before and after the cognitive module as a pre-and post-test. The difference between the pre- and post-test scores was used to assess the knowledge gained as a result of the didactic component.
Assessment of the technical and non-technical skills
Two of the authors NA and MA have developed a new assessment tool named Simulation Endoscopic Skill Assessment Score (SESAS) [Table 1] in January of 2018. Another author, NK has reviewed the task deconstruction of the procedure. SESAS came as a modification of the Direct Observation of Procedural Skills (DOPS) tool that has been developed by the Joint Advisory Group (JAG) for gastrointestinal endoscopy in the United Kingdom. DOPS is a global assessment tool for gastrointestinal endoscopic skills and has been used by the Saudi fellowship committee to evaluate the psychomotor skills of the trainees for a few years. However, it lacks comprehensive competency assessment, with little procedure-related cognitive skill assessment. Even though different programs are using this assessment tool worldwide, up to our knowledge, it has not been validated as a benchmark for competency thresholds and the trainees can be scored as passed, even if some simulator benchmarks were not achieved. The new assessment tool SESAS was developed using task deconstruction where we have put our efforts to improve the profession's quality metrics and competency assessment for diagnostic upper GI endoscopy. All the experts who participated in the curriculum-design validated the content of the newly developed checklist through modified Delphi consensuses using the face-face followed by two email rounds methodology  that we have previously described for the curriculum content validation. [Table 4] shows each checklist task and its relevant VRS metrics.
|Table 4: Each SESAS Checklist Task and its Equivalent from the VRS Metrics|
Click here to view
Technical, perceptual and decision making components of the non-technical skills were assessed during every simulation exercise using the SESAS and the relevant simulator metrics. The proficiency benchmarks for each task were based on the mean of the scores of the two trials performed by the five experts. The set level of proficiency for novice trainees was two standard deviations below the mean of the experts as per the methodology described by Gallagher AG and O'Sullivan. Dialogic bi-directional feedback on communication, ethical behavior of the patient and professional and continuous development was provided to the trainees in a debriefing session following the simulation. Outcome measures and metrics of these non-technical components are mentioned in [Table 2].
Descriptive statistics were computed for continuous variables, including minimum and maximum values, means, standard deviations, as well as 95% confidence intervals (CIs) and frequencies for categorical variables when appropriate. The Pearson's chi-squared test, t-test and, where appropriate, the Fisher's exact test were used.
R Studio was used for analysis using the R statistical language. A statistical significance threshold of P = 0.05 was adopted.
Cohen's kappa coefficient was used to assess the degree of agreement between the novel checklist SESAS and metrics generated by VRS and it was used as well to assess the inter-rater reliability between SESAS and DOPS.
| Results|| |
A total of 5 experts and 10 inexperienced GI fellows were recruited for the current study. All experts met the criteria indicated in the methodology section. The characteristics of the participating experts are shown in [Table 5].
All 15 participants answered the 10 MCQs. The mean score of the 5 experts was 9.0. All of the 10 inexperienced trainees completed the baseline MCQs pre-test. The mean score for the pre-test for this group was 2.7. After attending the cognitive module lectures, videos and hands-on workshop, all the inexperienced trainees re-answered the same MCQ as a post-test. The median MCQ pre- and post-test scores were 2.7 vs. 7.8 (P < 0.01).
Endoscopic technical and non-technical skills
The differences in performance between experienced and inexperienced (novice) endoscopists were compared.
The inexperienced trainees took significantly longer procedure time and time to pylorus compared with the experts (P < 0.01) although they were better than the experts in intubation of the esophagus with patient swallowing (P = 0.026) and amount of air left in stomach (P < 0.001). On the other hand, the experts needed a statistically significant smaller number of attempts to intubate the esophagus (P = 0.05). After training, there were no statistically significant differences between both trainees and experts regarding the rate of major complications (tracheal intubation, laryngeal edema), identification of landmarks, and patient discomfort. [Table 6] demonstrates that the novices have reached most of the proficiency benchmarks as compared to experts.
|Table 6: Comparison between the Two Trials' Performance of Experts and Trainees Using Metrics Generated by Virtual Reality Simulator (VRS) a. Time Metrics (min.)|
Click here to view
The following are the simulator metrics that showed evidence of construct validity (baseline discriminative ability between novices and experts):
(1) Time: total procedure time, time to upper esophageal sphincter intubation, time to pylorus intubation, time in red-out
(2) Esophageal intubation: number of attempts to intubate the esophagus.
(3) Occurrence of perforation as a complication.
Validation of the novel checklist
A total of 10 novices were assessed for their endoscopic skills at the end of the training using the new checklist (SESAS) by two independent evaluators and the degree of agreement between the new checklist and metrics generated by the VR simulator was found to be high (kappa = 0.83) [Figure 2].
|Figure 2:Comparison of performance of experienced and inexperienced endoscopists on VRS tasks and SESAS which was done at the end of training|
Click here to view
The degree of agreement and inter-rater reliability between SESAS and DOPS was found to be excellent (kappa = 0.90) [Figure 3].
|Figure 3:Comparison of performance of experienced and inexperienced endoscopists based on DOPS and SESAS at the end of training|
Click here to view
| Discussion|| |
The need for a comprehensive, standardized curriculum for GI endoscopy for gastroenterology fellowship programs was identified resulting in the commissioning of a task force to develop a 'Fundamentals of GI Endoscopy' curriculum using simulation-based education. The primary outcome of the current study was to develop a FGE simulation course that can improve novice performance in GI endoscopy prior to patient encounter. The second outcome was to determine which of the VR simulator metrics demonstrate evidence of construct validity. This is in addition to demonstrating the efficacy of the newly developed checklist (SESEAS). These outcomes were demonstrated throughout the components of the curriculum – knowledge, technical skills, and non-technical skills using a proficiency-based progression (PBP) methodology.
In skills training, it is very important to assess patient safety, as measured by occurrence of critical errors. In the current study, no major critical error (e.g. perforation) was reported by any of the trainees following the technical skills training. Also, although experienced endoscopists intubated the oesophagus in less time and less attempts (on the first attempt), most of novices intubated the oesophagus within two attempts. Time is not a critical factor, and with experience, the novices will continue to improve intubation skills without jeopardy to patient safety. Of major importance was the ability to successfully enter and examine the stomach and duodenum, and all were able to enter the pylorus and identify the Papilla of Vater (one of the hallmarks of a successful upper endoscopic procedure) by the completion of training. Finally, but of expected minor consequence, the novices have actually evacuated more air upon completion of the procedure than experienced endoscopists. This theoretically results in less post procedural patient discomfort.
So, the current single-arm study demonstrates that novices trained on this curriculum can reach a pre-set proficiency benchmark level of safe performance. As such, the trainee is expected to be able to perform safely an endoscopic procedure on a true patient under an experienced endoscopist's minor active then passive supervision. Despite the claim that achieving proficiency in simulation-based training constitutes “prima fascie” evidence for the competency of the novice, we are currently planning for a follow up study at the workplace to collect evidence for consequences validity.
Our new checklist-based global assessment tool, (SESAS), that was developed based upon and compared to the internationally used DOPS checklist, has demonstrated comparability to the novices' performance results that are automatically generated during training on the AccuTouch® GI endoscopy simulator. We hope that providing the simulation-based education community with a global assessment tool, that incorporates but does not solely rely on simulator outcome measures and metrices of construct validity, would lead to more objective assessment of trainee performance.
Strengths of the current study include: (1) following a validated stepwise approach for simulation-based curriculum development that is reached by international consensus  and is currently being used for curriculum development in multiple countries, (2) providing a model for the incorporation of simulator metric outcome measures and metrics of proved construct validity for objective assessment. Limitations to the study include the small number of participants and whether these results would transfer to other available GI simulators. Replication of these results by other independent researchers would strengthen the efficacy of the curriculum. Follow up studies for the transfer of competencies of the simulation trained fellows to the workplace are highly recommended.
| Conclusion|| |
The 'Fundamentals of GI Endoscopy' curriculum is a comprehensive, proficiency-based, standardized endoscopic training curriculum that has been primarily validated and can serve as a valuable addition to the gastroenterology fellowship programs. The evidence reported herein supports the effectiveness of the simulation curriculum for learning, the reliability and validity of the new SESAS as an assessment tool, and the importance of using the metrics-based methodology to engender patient safety. Workplace follow up study of trainee performance is recommended for consequences validation.
Financial support and sponsorship
This project was funded by an educational grant from the Saudi Gastroenterology Association.
Conflicts of interest
There are no conflicts of interest.
| References|| |
MacSween HM. Canadian Association of Gastroenterology Practice Guideline for granting of privileges to perform gastrointestinal endoscopy. Can J Gastroenterol 1997;11:429-32.
Ziv A, Ben-David S, Ziv M. Simulation based medical education: An opportunity to learn from errors. Med Teach 2005;27:193-9.
McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med 2011;86:706-11.
Ekkelenkamp VE, Koch AD, de Man RA, Kuipers EJ. Training and competence assessment in GI endoscopy: A systematic review. Gut 2016;65:607-15.
Walsh CM, Sherlock ME, Ling SC, Carnahan H. Virtual reality simulation training for health professions trainees in gastrointestinal endoscopy. Cochrane Database Syst Rev 2012;CD008237. doi: 10.1002/14651858.CD008237.pub2.
Singh S, Sedlack RE, Cook DA. Effects of simulation-based training in gastrointestinal endoscopy: A systematic review and meta-analysis. Clin Gastroenterol Hepatol 2014;12:1611-23.e4.
King N, Kunac A, Merchant AM. A review of endoscopic simulation: Current evidence on simulators and curricula. J Surg Educ 2016;73:12-23.
Zupanc C, Burgess-Limerick R, Hill A, Riek S, Wallis GM, Plooy AM, et al.
A competency framework for colonoscopy training derived from cognitive task analysis techniques and expert review. BMC Med Educ 2015;15:216.
Eversbusch A, Grantcharov TP. Learning curves and impact of psychomotor training on performance in simulated colonoscopy: A randomized trial using a virtual reality endoscopy trainer. Surg Endosc 2004;18:1514-8.
Aggarwal R, Ward J, Balasundaram I, Sains P, Athanasiou T, Darzi A. Proving the effectiveness of virtual reality simulation for training in laparoscopic surgery. Ann Surg 2007;246:771-9.
Ferlitsch A, Schoefl R, Puespoek A, Miehsler W, Schoeniger-Hekele M, Hofer H, et al
. Effect of virtual endoscopy simulator training on performance of upper gastrointestinal endoscopy in patients: A randomized controlled trial. Endoscopy 2010;42:1049-56.
Sedlack RE, Coyle WJ, Obstein KL, Al-Haddad MA, Bakis G, Christie JA, et al.
ASGE's assessment of competency in endoscopy evaluation tools for colonoscopy and EGD. Gastrointest Endosc 2014;79:1-7.
Khamis N, Satava R, Alnassar S, Kern D. A stepwise model for simulation-based curriculum development for clinical skills, a modification of the six step approach. Surg Endosc 2016;30:279-87.
Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Manag Sci 1963;9:458-67.
Gallagher AG, O'Sullivan GC. Fundamentals of Surgical Simulation: Principles and Practice (Improving Medical Outcomes – Zero Tolerance). London: Springer Verlag; 2012.
Khamis N, Satava R, Kern D. Stepwise simulation course design model: Survey results from 16 centers. JSLS 2020;24:e2019.00060.
Dr. Majid Almadi
Division of Gastroenterology, King Saud University Medical City, Riyadh - 12372
Kingdom of Saudi Arabia
Source of Support: None, Conflict of Interest: None
[Figure 1], [Figure 2], [Figure 3]
[Table 1], [Table 2], [Table 3], [Table 4], [Table 5], [Table 6]
| Article Access Statistics|
| Viewed||985 |
| Printed||31 |
| Emailed||0 |
| PDF Downloaded||130 |
| Comments ||[Add] |