P Schick, S Abramson, E Gnall, S Fine, J Burke
P Schick, S Abramson, E Gnall, S Fine, J Burke. The Value Of The Post-Hoc Analysis Of Responses To Questions Asked By Audience Response Technology. The Internet Journal of Medical Education. 2012 Volume 2 Number 2.
Context and background: There is a need to monitor the effectiveness of medical education. To meet this challenge, we pioneered an unappreciated application of audience response technology; the post-hoc analysis of answers to questions asked during lectures to monitor trainee knowledge and progress. . Procedure and results: Audience response technology is widely used to poll knowledge and opinions that engage audiences and promotes interactivity. Post-hoc analysis was feasible since the technology stores all answers (an underappreciated function of the technology). We used this approach in our cardiology fellowship conferences. The following information was obtained: 1) Baseline knowledge. This information helped us focus on poorly understood topics. 2) Two fellows had inadequate grades and required intervention. 3) Eleven questions were found to be ambiguous and were not graded. 4) We developed a strategy to assess the comprehension of essential core knowledge and ascertained that the fellows had a solid understanding of milestones. Impact and significance: This information was valuable for optimizing our training program. Fellow feedback: fellows strongly felt that audience response technology promoted interactivity and had educational value. In addition, grading by the technology was not intrusive, distracting or intimidating. Therefore, In comparison to stand-alone testing, monitoring knowledge by this technology was: 1) easier to implement, grade and evaluate; 2) more comprehensive and sensitive since all questions asked in the lecture series were assessed. The strategy was designed to meet the needs of our training program but could be adapted to the needs of any medical curriculum and training at all levels of medical education
Introduction – background
There is a need to improve the monitoring of the effectiveness of medical education programs. In order to meet this challenge, we pioneered an unappreciated application of audience response technology (ART); the post-hoc analysis of audience responses to monitor knowledge.
ART is being used in numerous medical centers and hospitals (1-6). Using this technology, medical students or trainees use remote control clickers to choose answers to questions displayed on PowerPoint slides. The group’s answers are instantly displayed, and this engages participants and promotes interactivity. This approach can improve educational programs and address the limitations of the passive nature of conventional lectures. It is generally believed that comprehension and retention of new material is improved if learners are actively engaged in the learning process.
ART stores all responses to questions. Therefore, it is possible to mine the stored data and carry out post-hoc analysis to monitor participant knowledge. This is an underappreciated feature of ART. An exhaustive literature search could not find any published information about the post-hoc analysis of ART responses. In a pilot study, we used this approach in two hematology lectures. The study provided useful demographic information about differences in knowledge in medical students and residents at different levels of training. (7).
We would like to report our experience using the post-hoc analysis of audience responses in a series of lectures in our cardiology fellowship program as an example of the value of this approach. The study demonstrates that post-hoc analysis of audience answers can meet the challenge of sensitively monitoring group (aggregate) and individual knowledge and progress. We have found that this educational approach is objective, easy to implement and grade, and has been enthusiastically accepted by trainees.
The program was designed to meet the needs of our training program. However, this technology can be tailored to the needs of any medical curriculum and training programs. Therefore, it would be of universal value at all levels of medical education.
1) Identification of lectures in which fellows achieved outstanding and poor grades: Well understood and poorly understood subjects were identified by mean grades that were one standard deviation above or below the mean grade of all lectures.
2) The evaluation of fellow grades: We compared fellow grades to determine whether any trainee needed additional help. Remediation was implemented for fellows whose grades were one standard deviation below the mean grade achieved by all fellows.
3) Identification of unfair and poorly designed questions: The faculty evaluated individual questions that fewer than 33% of fellows answered correctly. The purpose was to determine whether any of these questions were unfair due to inappropriate distracters, improper wording or other errors. If deemed unfair, the questions were not graded.
4) Use of post-hoc analysis to determine the comprehension of essential core knowledge (milestones). The fellowship director and two other academic cardiologists involved in the lectures identified questions about essential core information. Grades in these questions were recorded to determine comprehension of milestones.
Study was IRB exempt when reviewed by the Lankenau Medical Center IRB (Ethical Approval).
The instant display of responses engaged the fellows and promoted interactivity. Lecturers seeing the real-time display of their comprehension could spend more time on subjects shown to be poorly comprehended..
1) Identification of lectures in which fellows achieved outstanding and poor grades: The average grade of audience responses in 34 lectures was 72.8%
2) The evaluation of fellow grades: The data revealed that two fellows had grades that were one standard deviation below the mean grade of all fellows. This provided the faculty with an objective tool for the assessment of knowledge at a pivotal time point and to implement remediation and faculty intervention.
Identification of unfair and poorly designed questions:
3) The evaluation of questions: Eleven questions were found to be ambiguous due to inappropriate distracters, confusing wording or other errors. These questions were not graded.
4) Comprehension of essential core information (milestones): Fellows achieved a grade of 83.1 ± 5.3% (mean ± SD) (8).
The application of ART to our cardiovascular lecture series engaged participants who felt that the technology improved the lectures and promoted interactivity. The instant display of fellow knowledge also helped lecturers adjust their lectures by spending more time emphasizing poorly understood topics (4).
However, the main purpose of the current study was to present the value and the strategy of the post-hoc analysis of responses to questions asked by ART. The specific information about our training program was relevant to our institution only since all training programs differ in organization and strengths and weaknesses.
We were able to ask 1,293 questions in the lecture series using ART without sacrificing time spent on discussing the topics. There were several reasons why we were able to ask a large number of questions. The length of many lectures was greater than an hour, thus providing more time for questions. There were many questions on identification of abnormalities in ECGs, echocardiograms and pathology. These subjects were covered in the introductory lectures and fellows did not need much time to answer. The lecturer could still spend time discussing slides that fellows had misinterpreted, but residents correctly answered most of the slides and hence the lecturer was able to keep a fast pace.
There are several advantages of monitoring knowledge by ART by data-mining or post-hoc analysis of ART responses instead of relying on standard standalone examinations. Post-hoc evaluation was very comprehensive. The questions covered all points and topics presented in the lectures. Therefore, we were able to detect subtle differences in knowledge. Asking as many as 1,293 questions would not be feasible by standard testing. It was easier to implement testing, to grade, and to analyze fellows’ answers to questions by using audience response systems than by standard stand-alone examinations. When using ART, answers to questions were graded by a software program. Table 2 reveals that fellows thought that ART improved interactivity in lectures and that the technology was not intimidating.
Since the data about responses to questions asked in the lectures were transferred to MS Excel spreadsheets, we analyzed the data for information that we considered important to help monitor and improve our cardiovascular lecture series. However, other training programs might need additional or different types of information. Each training institution can tailor the post-hoc analysis to their needs to optimize their educational programs.
Testing knowledge using ART at this pivotal point had several merits. The documentation of fellow knowledge at the time point provided ample time to address deficiencies in knowledge prior to the end of the academic block. Fellows became aware of their strengths and weaknesses, and this encouraged self-study. Faculty awareness of fellow performance at this pivotal point permitted faculty intervention and remediation.
The cost of an ART license, software, and clickers would range from $1,500 to $3,000. However, most institutions have purchased ART and clickers. There would be no other costs except for an administrator or secretary to help carry out the analysis that would require 10 minutes per lecture.
The implementation of ART post-hoc analysis is feasible. Many institutions are using ART in the classroom without difficulty. Most good teachers have developed skills in designed questions and using PowerPoint to give lectures. Incorporating questions into ART enabled- PowerPoint lectures is an easy task. Successful post-hoc analysis is dependent on a designing a strategy for analysis and selecting and training an individual to perform the analysis using ART software and MS Excel. One of our administrators had no difficulty in learning these tasks and could analyze a lecture within 10 minutes. Conclusions:
Post-hoc analysis of the data provides detailed information of participant knowledge, progress and the mastery of milestones. This can enable program directors to be sensitive to strengths and weaknesses of training programs, identify a below average fellow performance, and to institute measures to improve education and the training program. Faculty development can be implemented when poor lecturers are identified and encouraged to improve. There are many other options for post-hoc analysis of ART. For example, if the audience is composed of medical students, residents and fellows, post-hoc analysis can reveal demographic differences in knowledge (7). This information can be used to tailor lectures and education programs to the demographic composition of the audience. In comparison to stand-alone standard testing, monitoring knowledge by ART was: 1) easier to implement, grade and evaluate and; 2) more comprehensive and sensitive since all questions asked in the entire lecture series could be assessed.
The specific information about our lecture series is relevant only for our fellowship program since the strengths and weaknesses and structure of each training program are unique. We have presented our data derived from ART as an example of the value of post-hoc analysis. However, the value and strategy is applicable to all educational programs and can be used at all levels of medical training and in all specialties of medicine.
A 500 word overview of the study has been published in Medical Education (Really Good Stuff) 45:1157-1158, 2011