Background

The emergence of digital technology has provided novel educational tools. Computer based medical education is an example of these technologies. There are a large number of computer-based tutorials on a wide range of medical topics. These tutorials vary in design and purpose. Computer tutorials have presented clinical simulations and real life presentation of interventive procedures and this approach has been a powerful educational tool (1,2,3). There are several merits of these programs. Medical students have been shown to learn more effectively and fell less intimidated by difficult concepts (4). When the program asks questions followed by instant feedback, learners who answered clinical questions correctly can opt to skip explanatory text. This “adaptive format” can shorten time spent on the computer tutorial (5. Computer based programs have been used to gather data for accreditation (6). A number of studies have demonstrated that computer tutorials are effective teaching tools (7,8). There are several obvious benefits of computer tutorials. The educational content of these programs can be standardized since their content can be peer reviewed. Participants can study computers at a time and place that is convenient for them. Despite their availability, computer tutorials are underutilized by medical educators (9,10). A roadmap to increasing the use of computer tutorials has been proposed. It includes the incorporation of multimedia and interactivity, documentation of competence, uniformity of instruction (10).

Based on the experience cited above, we have developed computer tutorials that were designed to challenge participants to integrate and apply knowledge to resolve real life problems. Topics covered in the programs were introduced with thought provoking questions that covered essential core knowledge. Instant feedback and the rationale for correct answers were provided to maximize educational value and to promote interactivity.

The study presents a novel use of computer-based tutorials to establish the demographics of baseline knowledge. This strategy for delineating the demographics of knowledge is demonstrated by presenting the results of a pilot study of the use of computer tutorials on bleeding disorders by residents and fellows. Our experience and merits of this approach are described. The information on baseline knowledge can be applied to optimize curriculum and training programs. Our approach can be used in computer tutorials for all levels of education.

Methods

Computer-Based Tutorial

There were three sections in the tutorial:

1) Section of pathophysiogy that contained primarily multiple-choice questions.

Figure 1

Table I

Figure 2

Figure 3

Table II

2) Section on choosing diagnostic tests. After brief case histories were presented, participants were asked to choose appropriate diagnostic tests from a list of 18 coagulation tests shown in Table Ia. The list was similar to laboratory test order forms used in hospitals. More than one test had to be chosen to diagnose several of the disorders. This type of question promoted decision making as if the participant was in a hospital environment, and therefore, was more challenging than multiple-choice questions. After participants entered their choices, correct and incorrect choices as well as the rationale for correct choices were instantly displayed. There were 11 case histories in this section. For example, if the case history suggested

hemophilia, participants should have chosen tests to diagnose hemophilia and distinguish it from a related disorder, von Willebrand disease.

2a) Subsection on diagnosis. Participants were asked review the case history and associated laboratory results in order to diagnose the underlying disorder. Once a diagnosis had been selected, the rationale for the correct diagnosis was shown.

3) Section on the selection of appropriate therapeutic options for managing commonly encountered bleeding disorders. After brief case histories were presented, participants were asked to choose appropriate management options from a list of 20 choices of current therapies for treating all types of bleeding disorders. The choices are shown in Table Ib. The options in Table Ib included most treatments options available in a hospital environment. Participants often had to choose more than one option for appropriate management. As mentioned above, this type of question was considerably more challenging than a multiple choice format.

After participants entered their choices, correct and incorrect choices and the rationale for correct choices were instantly displayed. There were 12 cases histories in this section. For example, if a patient had taken excessive amounts of Coumadin (an anticoagulant), the correct therapeutic options would have been withholding Coumadin, giving vitamin K and possible fresh frozen plasma and prothrombin complex concentrates.

Participant feedback

Posttest

Post-hoc analysis

Results

Baseline knowledge

Fellows achieved a grade of 62.8% ± 5% (SD) while residents achieved a grade of 51% ± 8% (SD) in baseline knowledge in the tutorial (p<0.02).

The analysis of performance of fellows in the subsections of the tutorial revealed that fellows did adequately in pathophysiology (74.6%) and did well in diagnosis (85.3%). However, fellows did poorly in choosing diagnostic tests (52.8%) and appropriate therapies (54.6%). These data and their significance are shown in Tables IIIa.

Figure 4

Table IIIa

The mean and SD are shown in Tables IIIa and IIIb. A T-Test was used to determine significance of the differences.

Figure 5

Table IIIb

The analysis of performance of residents in the subsections of the tutorial revealed that they did poorly in pathophysiology (55%) and marginally in diagnosis (64.9%). Residents did extremely poorly in choosing diagnostic tests (36.1%) and appropriate therapies (37.6%). These data and their significance are shown in Tables IIIb.

Fellows choose about five wrong choices per ten questions in the sections on selecting lab tests and therapy (Table IIIa). Residents choose about ten wrong choices per ten questions in the sections on selecting lab tests and therapy (Table IIIb). This should be considered in context since there were 18 possible choices for lab tests in each question (Table Ia). There were 20 possible choices for therapeutic options in each question (Table Ib). Therefore, there was little evidence for wild guessing.

The mean and SD are shown in Tables IIIa and IIIb. A T-Test was used to determine significance of the differences.

Posttest

Feedback

Time spent

Statistical analysis

Discussion

The analysis of baseline knowledge indicated that as expected fellows had higher grades than residents did in all subsections. Most striking was that both fellows and residents had weaknesses in choosing diagnostic tests management tests and management. The analysis of responses to individual questions revealed the demographics of weaknesses in certain disorders. For example, both residents and fellows had extreme difficulty managing certain bleeding disorders but were very capable of handling other disorders. In some cases, residents but not fellows had low grades in handling other bleeding disorders. This demonstrates that post-hoc analysis can provide detailed information about strengths and weaknesses.

The depth of information on specific weaknesses in knowledge and differences between fellows and residents was in part due to the formulation of questions that resembled real life decision situations rather than using multiple-choice questions.

This information is potentially useful in adapting curriculum to the needs of residents and fellows. For example, there should be a greater emphasis on indications for diagnostic tests and appropriate management. In addition, curriculum should be adapted to the needs of residents and fellows based data provided by the assessment of baseline knowledge. .

The poor performance in the ability of both fellows and residents to order appropriate lab tests and to select appropriate therapy most likely was due to lack of hands-on experience with ordering lab tests and decisions about management. Neither of the two institutions in which this study was performed has a Hemophilia center. Therefore, trainees would not have had extensive exposure to coagulation disorders.

Participants, particularly hematology fellows, achieved much higher grades for diagnosis than in skills for selecting diagnostic tests and therapies. The most likely reason for the discrepancy was the nature of the questions. The case history and relevant laboratory tests were available to participants when participants chose a diagnosis. Therefore, the ability to choose a correct diagnosis was dependent on the recall of the clinical and lab features of a disorder. In contrast, when participants were asked to choose lab tests and appropriate therapies, they were shown simulated clinical scenarios and were asked to choose tests from a typical laboratory order form. Only a few subtle clues were presented in the case history when participants were asked to choose diagnostic tests and therapy. Therefore answering these types of questions required problem solving abilities and decision-making and were more sensitive to weaknesses than the question on diagnosis.

Both fellows and residents demonstrated a significant improvement in the overall grades in the posttest. The posttest did not test for diagnostic skills since we wanted primarily to test skills for managing bleeding disorders. Analysis of the grades in the subsection in the posttest demonstrated that there was a marked and significant improvement in the ability to choose lab tests and appropriate management options. In contrast, grades did not improve in pathophysiology. The basis for this discrepancy is not clear. However, one reason might have been that multiple choice questions were asked about pathophysiology. Therefore, they may have been less challenging than questions on choosing lab tests and therapy. Since our focus was on improving clinical skills, the evidence that the tutorial helped participants choose appropriate tests and therapy was gratifying. We would like to assume that fellows and residents who studied the tutorial were better prepared to evaluate and manage bleeding disorders.

Participant feedback was enthusiast about the educational value and interactivity of the program. Participants rated the educational value of questions as 1.28 and the quality of feedback and interactivity as 1.34 out of 5 with 1 being the highest rating. It appeared that our efforts to produce thought-provoking questions that were more complex than multiple-choice questions and the feedback for the rationale for correct choices were successful. The effectiveness of the questions was in part because the questions were peer reviewed. This is an advantage of preparing a computer tutorial.

It was important that participant feedback indicated that they were not threatened by the questions asked during the tutorial. Establishing baseline knowledge during the tutorial has several advantages over stand-alone tests. Testing by the tutorial was easy to implement and avoided spending additional time to administer and grade a stand-alone test. The internet program was able to carry out sophisticated grading of participant responses. For example, the tutorial recorded correct as well as incorrect choices of diagnostic tests and therapies. Therefore, testing baseline knowledge during the tutorial was less stressful, easier to carry out and more efficient than using stand-alone tests.

Conclusions

This is the first study to carry out this type of in-depth post-hoc analysis. The information derived from this analysis is potentially valuable for adapting curriculum or a training program for the needs of students and trainees.

Although the study was a pilot study due to small number of participants, it clearly demonstrated the strategy and potential value of estimating the demographics of baseline knowledge while participants use a computer tutorial.

The sensitive to strengths and weaknesses in knowledge was in part due to the formulation of questions that resembled real life decision situations rather than using standard multiple-choice questions.

The strategy for preparing challenging tutorials and monitoring knowledge described in this paper should be applicable to teaching any academic subject and at all educational levels and for enhancing a variety of training programs.

Acknowledgements and IRB:

The study was supported by a grant from the Sharpe-Strumia Foundation of the, Main Line Hospitals, Bryn Mawr, PA.

IRB: The study was judged exempt by the IRB.