Are We Making Progress? One Medical School’s Assessment of an Evolving Integrated Palliative Medicine Curriculum
J Kitzes, S Lisa, S Kalishman
J Kitzes, S Lisa, S Kalishman. Are We Making Progress? One Medical School’s Assessment of an Evolving Integrated Palliative Medicine Curriculum. The Internet Journal of Pain, Symptom Control and Palliative Care. 2009 Volume 8 Number 1.
In 2007, Palliative Medicine became a recognized medical subspecialty 1 and since 1996 has been a mandated curriculum for medical schools by many national accrediting organizations. 2-5 Systematic reviews of palliative medicine curricula in medical schools document diverse, non standardized formats. 6-9 Although more U.S. medical schools are incorporating palliative medicine curriculum, in 2007 American Association of Medical Education(AAMC) surveys showed 20-23 % of all graduating medical students reporting palliative medicine, pain management and end-of- life care training as inadequate. 10, In addition, assessment of palliative medicine curriculum is uneven or lacking. 8
To support the above observation, in 2007, we conducted a PubMed(MEDLINE) MeSH search using the terms “End of Life Care” & “Hospice Care” & “Palliative Care” and “ Education “ for the dates of 2001-2006. Palliative medicine curriculum assessment tools and outcomes were noted in 35 articles. The strategies included: pre/post tests of knowledge, self assessed attitudes and skills and programmatic assessments; Multiple Choice Questions knowledge tests; personal reflections of student and physician mentors; student reflection papers; Objective Structured Clinical Examination (OSCE); and the author’s qualitative assessment. The palliative medicine curriculum and methods of assessment varied greatly and did not allow for comparisons across institutions.
In 2002-2004, the University of New Mexico School of Medicine (UNMSOM) implemented an evolving four year integrated palliative medicine curriculum embracing the UNMSOM’s desired “Six Core Competencies” (based on American College of Graduate Medical Education (ACGME) competencies): (1) Medical Knowledge, Integration and Critical Reasoning, (2) Patient Care, (3) Interpersonal and Communications Skills, (4) Professionalism, (5) Ethics and Self Assessment, and (6) Community and Systems Based Practice. This paper will address the assessment tools, findings and limitations of this evaluation process.
Integrated Palliative Medicine Educational Program
At the UNMSOM, the four year curriculum is divided into three Phases. Phase I, the preclerkship curriculum, lasts approximately 21 months and includes introductory and organ system blocks, clinical skills, research and rural community experiences. Phase II includes seven required “third year” clerkships (Family and Community Medicine, Internal Medicine, Neurology, Obstetrics and Gynecology, Pediatrics, Psychiatry, Surgery) over a 12-month period. Phase III (12 months) includes four required clinical rotations (e.g. ICU, Subinternship, Ambulatory Medicine and Rural and Community Practice) and electives.
The goals of an incremental integration of a palliative medicine curriculum into the existing UNMSOM curriculum are to promote student learning in:
Knowledge and skills base in palliative medicine
Communication skills regarding difficult end of life care decision making
The strategy to achieve these learning objectives was interdepartmental collaboration which facilitated the integration of cross cutting themes in existing courses and rotations. In addition to palliative medicine, examples of integrated crosscutting themes include: ethics, culture and diversity competency, and pain management. This networking systems approach has increased the probability of sustainable success.11 Appendix 1 summarizes the evolving integrated palliative medicine curriculum as of 2008.
To evaluate the palliative medicine curriculum, we collected data specifically targeted to the identified “palliative medicine” curriculum including Death Rounds in the third year clerkship. Death Rounds is a small group educational model that promotes student self reflection, meta cognition, professional growth, and collegial support around the clinical, ethical, legal, professional, cultural, and spiritual aspects of death. 17 Students in classes 2005-2007 completed three validated questionnaires. The three are: Concerns about Dying (CAD), 12 Physicians’ End -of-Life Care Attitude Scale (PEAS),13 and Competencies and Concerns in End-of-Life Care for Physician Trainees (CCEOLC).14 Other measures used were the UNMSOM Cross Cutting Themes Survey, Death Rounds Questionnaire (DRQ) 16,17 and the AAMC Graduation Questionnaire .. Table 1 summarizes the administration schedule of the above described assessment tools.
The Concerns about Dying Instrument (CAD)
The CAD is a brief self-report questionnaire to assess general concerns about death in health care providers .12 The CAD measures three factors through 10 statements: general concern about death, spirituality and patient-related concern about death. Each item is scored using a Likert scale format with five anchored rating choices from disagree completely to agree completely. In a validation study with third year medical students, test-retest reliability of the three scores was good (r = 0.84, 0.89, 0.83, respectively), internal consistency was adequate (alpha = 0.73, 0.76, 0.85, respectively), and correlations with related measures supported its construct. 15
The Physicians’ End-of-Life Care Attitude Scale (PEAS)
PEAS, a self reported instrument, measures the willingness of medical students /residents to interact with dying patients and their families, and was developed as an outcome measure for palliative care education. PEAS developers now recommend use of a 35 Likert-type item form version of PEAS that are matched into two subscales (Personal Communication, Hayslip, 2005). These subscales assess unique communication concerns of physician trainees in their interactions with dying patients. The personal preparation subscale with 16 items and the professional role subscale with 19 items are scored to indicate higher communication concerns with higher scores. The personal preparation subscale assesses personal feelings about interacting with dying persons and families including personal touch, conversation about end-of-life topics, feelings and comfort when speaking or being with dying patients or their families. The professional role subscale assesses the trainee’s difficulties in disclosing a poor prognosis to a patient; communicating with the patient about the projected future course of his or her illness; sharing or withholding information from the patient; discussing advanced directives with the patient; emotional boundaries in caring for dying patients; breaking bad news; and feeling inadequate as a professional when a patient dies. Excellent internal consistency and reliability are reported on the two subscales; validity studies with CAD subscales are also reported. 13
Competencies and Concerns in End-of-Life Care for Physician Trainees (CCEOLC)
This survey measures students/residents/fellows’ self-assessed competencies and concerns about end-of-life care. 14 We selected and modified three of the four domains in the survey: competence associated with six end-of-life communication skills, 10 end-of-life medical care topics and topics to be included in future end-of-life care education.
Death Rounds Questionnaire (DRQ) Data
Two tools assessed Death Rounds: a post DRQ and a Facilitators’ Log of observed student learning issues. The questionnaire was developed based on findings reported by Rhodes-Kropf et al, 2005 about medical students’ reactions to their most memorable deaths. 16 Specifically, the students choose adjectives that best describe their most recent death, with whom they discussed these deaths, their level of comfort with death, the value of Death Rounds, learning issues they encountered and how they addressed them, and their assessment of their palliative care skills before and after participation in Death Rounds. The facilitators for Death Rounds recorded the student identified learning issues in a compiled log.
UNMSOM Cross Cutting Themes Survey (CCS)
Students evaluate the quantity and quality of instruction and training they receive in 15 topics identified as cross cutting topics included in the UNMSOM integrated curriculum during Phase II and Phase III (Years 3 and 4). Students rate the quantity and quality of instruction for each topic on a 1-5 scale. Topics associated with palliative medicine include: end of life care, management of pain, communication skills and ethics and professionalism.
AAMC Medical School Graduation Questionnaire (AAMC GQ) Individual Reports, 2006- 2007
The AAMC GQ is completed by graduating students at Liaison Committee in Medical Education approved medical schools to assess the overall curricular experience. An annual report is prepared for each school aggregating the responses from its school as well as responses in all schools. Questions included in the 2006 and 2007 AAMC GQ address palliative medicine/pain management and end –of- life care. There are two different question types used. In one format, students are asked “Do you believe time devoted to your instruction in the following areas was inadequate, appropriate, or excessive?” In a second format, students are asked to “Indicate your level of agreement with the following statements” using a Likert scale format with 5 anchors from strongly agree to strongly Disagree. The questions under Communication Skills begin with the following lead: “I am confident that I have the appropriate knowledge and skills to” “discuss treatment options with a patient with terminal illness” (2007 only) and “initiate discussion of DNR orders with a patient or family member.” An individual range, mean and respondent count data are reported to the individual school for the past three years and the current year.
Concerns About Dying (CAD)
The questionnaire was given to students to complete at the end of Year 2 (Classes of 2007-2009), 185 students responded, and at the end of Year 3 (Classes of 2006-2008), 191 students responded, which is a 75-85% return rate from each administration.
Overall demographics and response rate showed fifteen percent of survey respondents did not provide information on their age, gender and ethnicity. Of those who responded to the demographic questions, 64% were female, 50% ranged in age from 21 to 26, 30% ranged in age from 27 to 31, 63% identified as Anglo, 24% identified as Hispanic , 3% identified as Native American , and 0.5% identified as African American . Eighty to ninety percent of each class responded to the ten items on the CAD survey at each administration. More than 75% of the students were able to agree or disagree with each statement rather than choosing “neutral” for a response. Subscale scores spanned the entire range of possible scores from 1 to 5. In Results are shown in Table 2. Students’ general and spiritual concerns about death did not change over the year they participated in the palliative medicine curriculum. However, their concerns about working with dying patients decreased.
Physicians’ End-of-Life Care Attitude Scale (PEAS)
Sixty-three students from the Class of 2006, and sixty-two students from the Class of 2007 responded to the PEAS questionnaire (80-85% ) during their third year. We did not include analysis from respondents in the first two years of medical school because they used the “not applicable” (NA) option for several items and their total and subscale scores gave incomplete information. As seen in Table 3, average scores on questions in both the Professional Role and Personal Preparation subscales suggest that at the end of the third-year clerkships, students’ communication apprehension is low, with no significant difference reported between the two classes. Third year UNMSOM students in Class of 2006 and 2007 participated in the fully integrated palliative medicine curriculum.
Competencies and Concerns in End-of-Life Care (CCEOLC)
Sixty-seven of 132 (51%) graduating medical students in the Classes of 2005 – 2006 filled out the survey. Forty-one students (53%) in the Class of 2007 responded to a modification of the survey. A significantly lower percentage of students in the Class of 2005 (42%) rated themselves competent to perform, independently or with minimal supervision, the 6 end-of-life communication skills listed in Table 4 than did students in the combined Classes of 2006 and 2007 (67%). A significantly lower percentage of students in the Class of 2005 (33%) also rated themselves competent to perform, independently or with minimal supervision, care related to the 10 end-of-life topics listed in Table 4 than did students in Classes of 2006 and 2007 combined (60%). In addition, in Table 5, over 85% of students, feel the need to learn more about 10 topics in future educational programs.
Death Rounds Questionnaire (DRQ)
Ninety percent (90%) of the students experienced a recent patient death during a clerkship. They selected diverse adjectives to describe the impact of the experience in personal terms. Our findings about how students describe their encounters with a patient’s death are consistent with previous results reinforcing that students have strong emotional responses to patient deaths. About one quarter of our students spoke about the death with an attending physician. A majority of students found Death Rounds to be an educational model that increased their comfort level about death and management of the dying process. 17 One student who had not experienced a patient death commented, “ I think it’s [Death Rounds] a good idea. I only wish I had known patients that died so I could deal with these important issues. I had several “bad news” scenarios, but no deaths in any patient that I met.”
Evidence in both the DRQ and Facilitators’ Logs demonstrate that multiple topics were addressed and were associated with the School of Medicine’s six core competencies. Students’ self ratings of knowledge, skills and competence after Death Rounds supported this evidence. 17
UNMSOM Cross Cutting Themes Survey (CCS)
Ninety to ninety-five percent of the students in Phase II and Phase III given the CCS in the Classes of 2006- 2008 completed it. As seen in Table 6, all mean scores for both quantity and quality of training on Cross Cutting issues associated with palliative care were at or above a mean of 3.0 on a five point scale except for Management of Pain.
AAMC Graduation Questionnaire, 2006 and 2007 Report
Ninety-three percent of the students in Class of 2006 and 96 percent of the students in the Class of 2007 from UNMSOM completed the AAMC GQ.
In response to the question, “Do you believe time devoted to your instruction in the following areas was inadequate, appropriate, or excessive?”, 65.6% in 2006 and 82.9%
in 2007 of the UNMSOM completing the questionnaire rated instruction in palliative care/ pain management as appropriate. Included in this report is the percentage from all
respondents across AAMC sites. In the All School 2007 rating for this item, 75.8% of students reported receiving appropriate instruction in palliative care/pain management. In response to a similar question about time devoted to instruction in End of life care, 63.9% in 2006 and 76.3% in 2007 of the UNMSOM students rated instruction as adequate, and 78% of 2007 students from All Schools rated instruction as appropriate.
Eighty-eight percent of 2007 UNMSOM students and an identical percent of 2007 students from All Schools agree or strongly agree with the statement, “I am confident that I have the appropriate knowledge and skills to discuss treatment options with a patient with terminal illness”. Eighty-seven percent of 2006 UNMSOM students and 88 percent of 2007 UNMSOM students agree or strongly agree with the statement, “I am confident that I have the appropriate knowledge and skills to initiate discussion of DNR orders with a patient or family member.” Eighty-five percent of 2007 students from All School agree or strongly agree with this statement as well. These findings echo the significant improvements documented throughout the United States from 1998-2006. (18)
Each item was rated on a 1-5 scale where a rating of 4 or 5 indicates concerns about death or discomfort working with patients who are dying. Subscale scores are averages of the ratings of all items in the subscale.
The objectives of the palliative medicine curriculum are to increase students’ palliative medicine knowledge and skills, and abilities to work with severely ill and dying patients. We selected tools focused on students;1) attitudes (PEAS, CAD), 2) perceptions of the curriculum (AAMC GQ, CCEOLC, and CCS), and 3) a mixed Death Rounds questionnaire. During this study, negotiated attempts to include palliative medicine OSCEs and multiple choice content questions in the preclinical phase were minimally effective due to the already packed assessment schedule and therefore, data is not available..
The overall palliative medicine curriculum assessment demonstrates a positive increase in students’ confidence, attitudes, and desire for more education. Students, who lack preclinical experience, as evidenced by PEAS data, are unwilling to answer many of the questions. However, after students have interacted with clinical patients and have some experience they respond.
Students’ reduced anxiety around communication issues in PEAS (Table 3) is an important outcome. Reducing student anxiety in end-of-life care communication may be related to student self- efficacy. Belief in one’s ability and knowledge to care for dying patients, interact with their families or console/ talk with team members when patients die is positively linked to subsequent student performance and goal attainment in several studies.19-22 Faculty support through encouragement or feedback, role modeling and the expressed belief of faculty in learners’ abilities to master the skills and knowledge associated with palliative medicine are essential self-efficacy components. 23
After participating in the evolving palliative care curriculum, students rated themselves more competent to perform independently or with minimal supervision the 10 end-of-life medical care topics.Additionally, similar to previous national studies, over 85% of our students, on average, requested more palliative medicine education. 6-8
Our study has several potential limitations. First, we are acutely aware that the evaluation time frame was very brief, and included data with only a few cohorts per instruments. Second, self-assessment data applied to generic knowledge and skills are generally not considered sufficient to state that the students gained the focal competencies solely from participation in the palliative care curriculum and can be overestimated by the students. 24 To assess students’ competence in palliative medicine knowledge and skills, we recognize the need to develop both “knows how and shows how” assessment tools. While we have developed several standardized patient cases to assess communications skills and symptom assessment/management, as well as palliative medicine multiple choice questions, they have not yet been implemented with UNMSOM medical students.
Finally, the tools used in this study reflect the diverse, non standardized attempts of palliative medicine medical educators to evaluate the developing curriculum hindering comparisons with other published studies. The tools described in this article facilitate assessment, but cannot stand alone due to their focused outcomes.
The assessment of the UNM SOM integrated palliative care curriculum has demonstrated progress towards enhancing students’ self- efficacy and confidence in palliative care knowledge, skills, and attitudes. Used alone, the evaluation tools are helpful but observed skills and knowledge acquisition measures are also recommended. By giving students exposure to authentic palliative medicine interactions (as evidenced in Death Rounds), they indicate they want more education. These interactions with real patients/families through the palliative medicine prism enhance the practice of medicine to encompass care beyond diagnosis and cure, and provide an anchor for the students’ palliative medicine knowledge they obtained in the preclinical years. In order to achieve further substantial palliative medicine education gains, we endorse the recommendation for academic centers to increase the number of palliative medicine specialists as role models through the development of academic palliative medicine clinical services and departments. 25 The UNM SOM will be initiating a Clinical Palliative Medicine Service and the development of a future Palliative Medicine fellowship training program in 2009. Hopefully, ongoing assessment will demonstrate improved end-of-life care for patients and enhanced palliative medicine experiences for our students.
We thank Jan Mines MA for her data analysis expertise. We also thank the Soros Foundation and Arthur Vining Davis Foundation for their support in the development, implementation and assessment of the evolving palliative medicine curriculum.