Introduction

Injury to the urinary tract during gynecologic surgery is not uncommon. In some series the ureteral injury rate is as high as 26.8/1000, with bladder injury at 29.1/1000¹. Diagnostic cystourethroscopy at the time of difficult gynecologic cases or cases where ureteral integrity is questioned has been shown to reliably identify ureteral injury^2,3,4. As the population ages it is likely that the number of cases for stress urinary incontinence, and pelvic organ prolapse will increase⁵. Currently, training in diagnostic cystourethroscopy of gynecology residents varies across the nation.

We designed an Objective Structured Assessment of Technical Skills (OSATS) using a bladder model, constructed out of a rubber ball and balloon, for teaching requisite skills necessary in the performance of diagnostic cystourethroscopy. The purpose of this study was to determine whether an educational program, assessed using the OSATS, improved the surgical skills necessary for the performance of diagnostic cystourethroscopy. Our hypothesis was that the use of this model and education provided in association with the use of this model would improve surgical skills compared to the group that did not receive training using the model.

Materials and Methods

We invited all residents of both obstetrics and gynecology training programs at The University of Texas Health Science Center at Houston (UTHSCH) to participate in the study. We fashioned a low cost bladder model out of a rubber ball and balloon. The rubber ball was approximately 4 cms in diameter, and in the center of the ball we created a 5mm urethra with the drill. The balloon fit snugly over the rubber ball, as depicted in the figure, and we attached the balloon and ball to the simulation table with a Velcro strap. We marked two areas in the balloon representing the ureteral orifi based on published anatomic criteria⁶, and also an area marking the dome of the bladder.

We created a written questionnaire and 11 point task specific checklist (Figure 1) in regard to diagnostic cystourethroscopy. We created the checklist based on essential steps necessary to perform a diagnostic cystourethroscopy and those highlighted in a contemporary text⁷. The written questionnaire assessed the learners’ knowledge in regard to diagnostic cystourethroscopy.

Figure 1

Figure 1: Task Specific Checklist

We divided the participants into an intervention group, which consisted of a one hour interactive presentation followed by two hours of practice using the bladder model, and control group, which received the usual education and teaching. The usual education and teaching was that performed in the operating room during diagnostic cystourethroscopy. This training likely varied based on the attending physician supervising the case but was performed on all cases in which there was the possibility of ureteral injury. Resident physicians were the primary surgeons on all cases performed in the operating room. The residents were divided by the first author of the study based on post graduate year of training. Prior to dividing the residents into groups the residency directors were polled to determine the number of diagnostic cystoscopic procedures performed, and no statistical differences were noted between participants.

We obtained informed consent of all participants and then performed baseline testing of all participants. We assessed knowledge with the written questionnaire. We then timed and evaluated the participants while performing a diagnostic cystourethroscopy on the bladder model using the global rating scale with pass/fail grade⁸ (Figure 2), and task specific checklist. We allowed the participants ten minutes to complete the task on the bladder model. The global rating scale consists of a pass/fail grade and a five point likert scale with seven categories: respect for tissue, time and motion, instrument handling, knowledge of instruments, flow of operation, use of assistants, and knowledge of specific procedure. The global rating scale is a validated scale to measure resident surgical performance. The task specific checklist, which along with the global rating scale comprised the OSATS, was not validated.

Figure 2

Figure 2: Global Rating Scale

Each participant had two evaluators, and four evaluators total participated in the study. Prior to participating in the study the evaluators met to discuss the evaluation tools so as to be uniform in evaluation interpretation. We deemed failure of the task as not being able to complete the task within ten minutes. All of the evaluators in the study were blind to group allocation except the first author.

We performed the baseline testing in August and November of 2008. We provided the intervention in December 2008, or usual teaching, followed by post intervention testing. We performed post intervention testing in the same manner as baseline testing in February and May of 2009. The participants were not blind to group assignment.

We obtained data in four categories at each evaluation: the written questionnaire, task specific checklist, global rating scale ranging from one to five, and the time to complete the task with a maximum of 600 seconds. We recorded the pass/fail portion of the global rating scale.

Statistical analyses were performed using the SAS statistical package. Inter-rater reliability was measured using the Pearson correlation coefficient test for continuous variables. Chi square and Fisher’s exact test when appropriate were performed to compare nominal variables. The Wilcoxon sum test was used to compare the continuous variables between cohorts, and the Wilcoxon matched-pairs signed rank test within cohorts.

We obtained IRB approval for the study from UTHSCH and report the data using the CONSORT guidelines for randomized controlled trials⁹.

Results

Forty one out of 45 (91.1%) residents agreed to participate in the study. Twenty-one residents were assigned to the intervention group and 20 to the usual teaching. Twelve participants in the intervention group and 13 in the control group had complete information with pre and post intervention measurements. Eight of the 12 residents assigned to the intervention group received the intervention (66.7%). Figure 3 is a flowchart demonstrating participant allocation.

Figure 3

Figure 3: Flow chart

The price of the rubber balls and balloons were negligible, and the drill and drill bits used to create a ‘urethra’ in the rubber ball about $40.

There were no statistically significant differences in postgraduate year of training between the residents in the intervention group and the control (p=0.93).

Participants were analyzed in the group to which they were assigned. We discovered no statistically significant differences in pre test or post test scores between those who were assigned to the intervention and those who were not. Table 1 lists the scores on the task specific checklist, global rating scale, questionnaire, and time to complete task at baseline (pretest) and after intervention (post test). We also found no significant differences in pass/fail grade at baseline or after intervention between the control and intervention group.

Figure 4

Table 1.

Table 2 outlines the results within cohorts and noted a statistically significant improvement from the pretest to post test within the intervention group in regard to the task specific checklist (.004), global rating scale (.004), time to complete in seconds (.006), and questionnaire (.01). We noted a statistical significant difference in the control group in the task specific checklist (.03) and a trend toward improved performance in the other tasks but this did not reach statistical significance.

Figure 5

Table 2.

We also analyzed the difference in improvement (pretest to post test scores) between the intervention and control group for the task specific checklist (mean 0.25 vs 0.19, p=0.549), global rating scale (mean 1.55 vs 0.63, p=0.057), and reduction in time to complete the task (mean 162 secs vs 85 secs, p=0.276). Although there was a trend toward improved performance in all areas measured there was no statistical significance achieved. The global rating scale was close to statistical significance.

Inter-observer reliability on pre/post test results for the task specific checklist (0.78); global rating scales (0.87) and pass/fail grade (1.0) were high.

Discussion

Surgical education has changed over the last two decades. As early as the 1970’s, surgeons had developed task specific checklists in order to evaluate operative skills¹⁰. With the work hour restrictions, emphasis from the ACGME to implement standardized teaching for surgical skills, and a move toward more competency-based surgical education, simulation learning is a necessary tool in resident education. Recent literature in regard to obstetric and gynecologic surgical simulation training would support its efficacy^11,12,13. Also, given the likelihood that the number of procedures that require diagnostic cystourethroscopy to ensure ureteral/bladder integrity will increase, we felt an educational program to teach this skill was necessary.

There are a number of strengths to our study including those associated with the use of a model versus observation of real time surgery. Our model is inexpensive. This is in contrast to a similar study used to evaluate diagnostic cystourethroscopy skills among residents where the costs associated with the model neared $4,000¹⁴. We had no operating room time costs. We avoided patient safety issues, and the model was standardized; the trainee was able to complete the task without concern for any patient related factor that would require an attending physician to step in and complete the task. Also, the time involved in the interactive session was not prohibitive, an hour for didactics and a couple of hours for practice.

Much like most of the gynecology simulation literature, we did not compare performance on the model in relation to live operating room performance. We cannot say whether superior performance on the bladder model leads to improved technical performance in the operating room. We believe the statistically significant increase observed in the control group in regard to performance on the task specific checklist and the trend toward improved overall performance can be explained by a number of factors; recent attention in the current literature to the importance of diagnostic cystourethroscopy, increase in the number of diagnostic cystourethroscopies performed at our institution during the study period, and involvement in the study itself may have motivated control group participants to better their performance (Hawthorne effect).

Given the resident rotation schedules, participants were not truly randomized. However, we doubt this was a source of bias as the numbers of senior and junior residents in each arm were similar, as evidenced by no statistical differences in scores at baseline testing. Participation was initially strong but waned due to a conflict in the laboratory space which changed the post intervention testing dates. We did not monitor the number of cystoscopes performed by study participants, possibly affecting post intervention scores. Given the low numbers, any significant conclusions proved difficult. Also, the evaluators knew the participants possibly introducing bias into the scores.

We observed those that were randomized to the intervention had statistically significant improvements in all evaluation categories, while the control group only showed an improvement in the task specific checklist. We believe that training using the balloon model and didactics that we have outlined improves the skills necessary for diagnostic cystourethroscopy.

We believe this economical model is useful in training residents in diagnostic cystourethroscopic procedures. A larger sample size is required to show if training improves skill level. We have incorporated this OSATs with bladder model into our residency curriculum.