Strategy for Enabling Computer-Based Tutorials to Document Baseline Knowledge in Hematological Subjects That Can Improve Curriculum and Training Programs
P Schick, M Burke
baseline knowledge, clinical scenarios, computer assisted instruction, interactive, problem-solving
P Schick, M Burke. Strategy for Enabling Computer-Based Tutorials to Document Baseline Knowledge in Hematological Subjects That Can Improve Curriculum and Training Programs. The Internet Journal of Hematology. 2009 Volume 7 Number 1.
The study presents a novel use of computer-based tutorials to establish the demographics of baseline knowledge. Thought-provoking questions were asked in the programs that introduced topics considered being essential information. Immediate feedback was provided to promote interactivity. Participant responses to questions were stored for post hoc analysis of baseline knowledge. This strategy for delineating the demographics of knowledge is demonstrated by presenting the results of a pilot study of the use of computer tutorials on bleeding disorders by residents and fellows. The study revealed that participant feedback was very favorable and indicated that the program had avoided a threatening testing environment. Computer tutorials can grade questions rapidly. Grading knowledge while participants use computer tutorials was more efficient and better accepted than grading by stand-alone testing. This information can be applied to optimize curriculum and training programs. Our approach can be used in computer tutorials for all levels of education.
The emergence of digital technology has provided novel educational tools. Computer based medical education is an example of these technologies. There are a large number of computer-based tutorials on a wide range of medical topics. These tutorials vary in design and purpose. Computer tutorials have presented clinical simulations and real life presentation of interventive procedures and this approach has been a powerful educational tool (1,2,3). There are several merits of these programs. Medical students have been shown to learn more effectively and fell less intimidated by difficult concepts (4). When the program asks questions followed by instant feedback, learners who answered clinical questions correctly can opt to skip explanatory text. This “adaptive format” can shorten time spent on the computer tutorial (5. Computer based programs have been used to gather data for accreditation (6). A number of studies have demonstrated that computer tutorials are effective teaching tools (7,8). There are several obvious benefits of computer tutorials. The educational content of these programs can be standardized since their content can be peer reviewed. Participants can study computers at a time and place that is convenient for them. Despite their availability, computer tutorials are underutilized by medical educators (9,10). A roadmap to increasing the use of computer tutorials has been proposed. It includes the incorporation of multimedia and interactivity, documentation of competence, uniformity of instruction (10).
Based on the experience cited above, we have developed computer tutorials that were designed to challenge participants to integrate and apply knowledge to resolve real life problems. Topics covered in the programs were introduced with thought provoking questions that covered essential core knowledge. Instant feedback and the rationale for correct answers were provided to maximize educational value and to promote interactivity.
The study presents a novel use of computer-based tutorials to establish the demographics of baseline knowledge. This strategy for delineating the demographics of knowledge is demonstrated by presenting the results of a pilot study of the use of computer tutorials on bleeding disorders by residents and fellows. Our experience and merits of this approach are described. The information on baseline knowledge can be applied to optimize curriculum and training programs. Our approach can be used in computer tutorials for all levels of education.
The computer-based tutorial on bleeding disorders was developed using Microsoft Visual Studio software. The tutorial was used by fellows in hematology training programs and residents in internal medicine training programs at Drexel University College of Medicine and Lankenau Hospital.
All topics were introduced by thought-provoking questions that covered essential or core knowledge necessary for diagnosing and managing hematological disorders. The questions had been formulated by the first author, a board certified hematologist, and peer reviewed by two additional board certified hematologists/oncologists. A key feature of the programs was that once the questions had been answered, the program instantly indicated whether the answers were correct and provided a rationale for the correct answer. Our questions simulated clinical scenarios as would occur in hospital emergency room and participants had to make decisions on management.
There were three sections in the tutorial:
1) Section of pathophysiogy that contained primarily multiple-choice questions.
2) Section on choosing diagnostic tests. After brief case histories were presented, participants were asked to choose appropriate diagnostic tests from a list of 18 coagulation tests shown in Table Ia. The list was similar to laboratory test order forms used in hospitals. More than one test had to be chosen to diagnose several of the disorders. This type of question promoted decision making as if the participant was in a hospital environment, and therefore, was more challenging than multiple-choice questions. After participants entered their choices, correct and incorrect choices as well as the rationale for correct choices were instantly displayed. There were 11 case histories in this section. For example, if the case history suggested
hemophilia, participants should have chosen tests to diagnose hemophilia and distinguish it from a related disorder, von Willebrand disease.
2a) Subsection on diagnosis. Participants were asked review the case history and associated laboratory results in order to diagnose the underlying disorder. Once a diagnosis had been selected, the rationale for the correct diagnosis was shown.
3) Section on the selection of appropriate therapeutic options for managing commonly encountered bleeding disorders. After brief case histories were presented, participants were asked to choose appropriate management options from a list of 20 choices of current therapies for treating all types of bleeding disorders. The choices are shown in Table Ib. The options in Table Ib included most treatments options available in a hospital environment. Participants often had to choose more than one option for appropriate management. As mentioned above, this type of question was considerably more challenging than a multiple choice format.
After participants entered their choices, correct and incorrect choices and the rationale for correct choices were instantly displayed. There were 12 cases histories in this section. For example, if a patient had taken excessive amounts of Coumadin (an anticoagulant), the correct therapeutic options would have been withholding Coumadin, giving vitamin K and possible fresh frozen plasma and prothrombin complex concentrates.
Participants were asked to answer questions in a questionnaire in the tutorial to obtain feedback on the value and efficiency of the tutorial. The questions asked are shown in Table II.
A computer-based posttest was given within 2 to 4 weeks after completion of the tutorial to assess the comprehension of topics discussed in the tutorial. Questions in the posttest covered topics discussed in the tutorial. The format of questions in the posttest was similar to that in the tutorial. Fifteen questions were asked: Five questions on pathophysiology, five questions to the ability to choose appropriate laboratory tests and five questions to assess skills in management. There were no questions on diagnosis since the posttest focused on management skills.
The unique feature of the study was the analysis of participant knowledge. Since the questions were asked prior to the discussion of the subjects, baseline knowledge could be determined. Participant answers to questions asked in the computer based tutorial were stored in the hospital intranet. Since participants could choose more than one diagnostic test and management option in the tutorial and posttest, the number of incorrect choices was also stored. Participants’ evaluations of the tutorial recorded in the questionnaire as well as time spent studying the tutorial were stored in the hospital intranet. The stored data could be retrieved and transferred to Excel spreadsheets for extensive post hoc analysis.
Fellows achieved a grade of 62.8% ± 5% (SD) while residents achieved a grade of 51% ± 8% (SD) in baseline knowledge in the tutorial (p<0.02).
The analysis of performance of fellows in the subsections of the tutorial revealed that fellows did adequately in pathophysiology (74.6%) and did well in diagnosis (85.3%). However, fellows did poorly in choosing diagnostic tests (52.8%) and appropriate therapies (54.6%). These data and their significance are shown in Tables IIIa.
The mean and SD are shown in Tables IIIa and IIIb. A T-Test was used to determine significance of the differences.
The analysis of performance of residents in the subsections of the tutorial revealed that they did poorly in pathophysiology (55%) and marginally in diagnosis (64.9%). Residents did extremely poorly in choosing diagnostic tests (36.1%) and appropriate therapies (37.6%). These data and their significance are shown in Tables IIIb.
Fellows choose about five wrong choices per ten questions in the sections on selecting lab tests and therapy (Table IIIa). Residents choose about ten wrong choices per ten questions in the sections on selecting lab tests and therapy (Table IIIb). This should be considered in context since there were 18 possible choices for lab tests in each question (Table Ia). There were 20 possible choices for therapeutic options in each question (Table Ib). Therefore, there was little evidence for wild guessing.
The mean and SD are shown in Tables IIIa and IIIb. A T-Test was used to determine significance of the differences.
Both fellow grades (62.8% ± 5% 81.8% ± 5% SD) (p = < 0.01) and resident grades (51% ± 8% 68.7% ± 10% SD) (p < 0.04) improved significantly in the posttest .
Tables IIIa and IIIb demonstrate that grades in subsections of the Post Test revealed that fellow and resident grades in pathophysiology did not significantly improve in the posttest. However, both fellows and residents grades in choosing lab tests and appropriate management significantly improved in the post test as shown in Tables IIIa and IIIb. Diagnostic skills were not assessed in the posttest since we wanted to focus on skills of managing patients. The significance of these data is shown in Tables IIIa and IIIb.
The programs were well accepted. Analysis of responses to queries in the questionnaire is shown in Table II. The data revealed that both fellows and residents thought that the program was very meritorious. The rating scale was 1 to 5 with 1 being the highest rating. The mean rating of both fellows and residents was 1.53. Therefore, the feedback grades were outstanding and comments were generally enthusiastic.
The tutorial also recorded time spent. Fellows spent an average of 69 minutes and residents 65 minutes using the tutorial. This appeared to be an acceptable amount of time and indicated that participants seriously applied themselves to using the tutorial.
Means and SD were determined. A T-test was used to determine the significance between groups of data.
The analysis of baseline knowledge indicated that as expected fellows had higher grades than residents did in all subsections. Most striking was that both fellows and residents had weaknesses in choosing diagnostic tests management tests and management. The analysis of responses to individual questions revealed the demographics of weaknesses in certain disorders. For example, both residents and fellows had extreme difficulty managing certain bleeding disorders but were very capable of handling other disorders. In some cases, residents but not fellows had low grades in handling other bleeding disorders. This demonstrates that post-hoc analysis can provide detailed information about strengths and weaknesses.
The depth of information on specific weaknesses in knowledge and differences between fellows and residents was in part due to the formulation of questions that resembled real life decision situations rather than using multiple-choice questions.
This information is potentially useful in adapting curriculum to the needs of residents and fellows. For example, there should be a greater emphasis on indications for diagnostic tests and appropriate management. In addition, curriculum should be adapted to the needs of residents and fellows based data provided by the assessment of baseline knowledge. .
The poor performance in the ability of both fellows and residents to order appropriate lab tests and to select appropriate therapy most likely was due to lack of hands-on experience with ordering lab tests and decisions about management. Neither of the two institutions in which this study was performed has a Hemophilia center. Therefore, trainees would not have had extensive exposure to coagulation disorders.
Participants, particularly hematology fellows, achieved much higher grades for diagnosis than in skills for selecting diagnostic tests and therapies. The most likely reason for the discrepancy was the nature of the questions. The case history and relevant laboratory tests were available to participants when participants chose a diagnosis. Therefore, the ability to choose a correct diagnosis was dependent on the recall of the clinical and lab features of a disorder. In contrast, when participants were asked to choose lab tests and appropriate therapies, they were shown simulated clinical scenarios and were asked to choose tests from a typical laboratory order form. Only a few subtle clues were presented in the case history when participants were asked to choose diagnostic tests and therapy. Therefore answering these types of questions required problem solving abilities and decision-making and were more sensitive to weaknesses than the question on diagnosis.
Both fellows and residents demonstrated a significant improvement in the overall grades in the posttest. The posttest did not test for diagnostic skills since we wanted primarily to test skills for managing bleeding disorders. Analysis of the grades in the subsection in the posttest demonstrated that there was a marked and significant improvement in the ability to choose lab tests and appropriate management options. In contrast, grades did not improve in pathophysiology. The basis for this discrepancy is not clear. However, one reason might have been that multiple choice questions were asked about pathophysiology. Therefore, they may have been less challenging than questions on choosing lab tests and therapy. Since our focus was on improving clinical skills, the evidence that the tutorial helped participants choose appropriate tests and therapy was gratifying. We would like to assume that fellows and residents who studied the tutorial were better prepared to evaluate and manage bleeding disorders.
Participant feedback was enthusiast about the educational value and interactivity of the program. Participants rated the educational value of questions as 1.28 and the quality of feedback and interactivity as 1.34 out of 5 with 1 being the highest rating. It appeared that our efforts to produce thought-provoking questions that were more complex than multiple-choice questions and the feedback for the rationale for correct choices were successful. The effectiveness of the questions was in part because the questions were peer reviewed. This is an advantage of preparing a computer tutorial.
It was important that participant feedback indicated that they were not threatened by the questions asked during the tutorial. Establishing baseline knowledge during the tutorial has several advantages over stand-alone tests. Testing by the tutorial was easy to implement and avoided spending additional time to administer and grade a stand-alone test. The internet program was able to carry out sophisticated grading of participant responses. For example, the tutorial recorded correct as well as incorrect choices of diagnostic tests and therapies. Therefore, testing baseline knowledge during the tutorial was less stressful, easier to carry out and more efficient than using stand-alone tests.
This is the first study to carry out this type of in-depth post-hoc analysis. The information derived from this analysis is potentially valuable for adapting curriculum or a training program for the needs of students and trainees.
Although the study was a pilot study due to small number of participants, it clearly demonstrated the strategy and potential value of estimating the demographics of baseline knowledge while participants use a computer tutorial.
The sensitive to strengths and weaknesses in knowledge was in part due to the formulation of questions that resembled real life decision situations rather than using standard multiple-choice questions.
The strategy for preparing challenging tutorials and monitoring knowledge described in this paper should be applicable to teaching any academic subject and at all educational levels and for enhancing a variety of training programs.
Acknowledgements and IRB:
The study was supported by a grant from the Sharpe-Strumia Foundation of the, Main Line Hospitals, Bryn Mawr, PA.
IRB: The study was judged exempt by the IRB.