M Kutcher, S Wakefield, K Mann, J MacCuspie, J Murphy, K Blake
M Kutcher, S Wakefield, K Mann, J MacCuspie, J Murphy, K Blake. Rating a Physician by Adolescents Using the Structured Communication Adolescent Guide (SCAG). The Internet Journal of Medical Education. 2009 Volume 1 Number 2.
The Structured Communication Adolescent Guide (SCAG) is a 29-item checklist used for teaching and assessing the adolescent clinical interview. The SCAG is divided into four sections (Getting Started, Gathering Information, Teen Alone [including lifestyles] and Wrap Up) each with a General Rating. The SCAG has been shown to be both reliable and valid when used by adolescents who have been trained in its use. We investigated the feasibility and reliability of the SCAG when used by Junior High School adolescents who had not been formally trained to use the SCAG.
The teaching and assessment of communication skills has become integral to the overall process of undergraduate and continuing medical education. Physician communication and interpersonal skills are a critical determinant of quality patient care.1, 2 Research illustrates that patient satisfaction,3 compliance,4,5 recall and understanding of treatment advice,6-8 disclosure of psychosocial information9,10 and attendance at scheduled appointments11,12 increase as a function of improved physician communication skills. Moreover, the importance of physician communication skills has been emphasized in guidelines and objectives for educational programmes,13-17, 39,40 consensus statements,18-20 and professional practice standards.21,22 Accordingly, effective communication skills are considered a core component of the clinical interview.
Typically, three methods are widely used for assessing communication and interpersonal skills: 1) checklists of observed behaviours during clinical interactions, 2) patient surveys, and 3) assessments of learning (oral, essay, multiple choice, OSCE).23 Checklists are the most commonly used of the above methods. For example, in the ten-year period from 1986 to 1996 more than 25 communication and interpersonal skills checklists were detailed in the communication-skills literature.24 Furthermore, educators have identified a need for age-specific communication instruments.25,26 This is particularly evident for adolescent patients.27
Medical students report low levels of confidence in interviewing adolescent patients, and that adolescent patients seem uniquely difficult to interview.28 This may be due to the fact that adolescents can occupy a variety of positions within the psychological, cognitive, and psychosocial domains.29 The adolescent years are a critical period in the development of behavioural traits associated with both positive and negative long-term health outcomes.30-32. In dealing with the presenting problem alone, physicians may fail to identify underlying reasons for the adolescents’ visit. How then to prepare medical students to conduct an effective adolescent interview?
Several years ago, two of the authors (KB and KM) studied the improvement of senior medical students’ adolescent psychosocial interviewing skills, and found that structured feedback from an adolescent and mother standardized patient (SP) led to significant improvements in the students’ performance.25 This study also served to highlight the need for a structured guide that could facilitate SP-delivered feedback. We developed our guide by adapting a well-known communication guide, the Calgary-Cambridge Guide.33 We called our tool the Structured Communication Adolescent Guide or SCAG28. The SCAG incorporates adolescent-specific psychosocial data-gathering along with communication skills that medical professionals are expected to use when conducting an adolescent clinical interview.27,29
In the years since it was first developed, the SCAG has been refined through an on-going research process that has involved adolescent input as a central component for evaluating its effectiveness. Currently the application of the SCAG extends well beyond a simple feedback guide, as research has shown it is a reliable and valid tool for trained adolescent standardized patients (SPs) to use in assessing medical students, residents, and physicians in practice.28,34 The SCAG is currently employed as a learning tool for medical students in their Paediatric clerkship at five Canadian Medical Schools, where it provides a framework for students to follow when conducting an adolescent interview.35
The SCAG is a 29-item communication and interview skills checklist, and is divided into four sections: 1) Getting Started, 2) Gathering Information, 3) Teen Alone [lifestyles], and 4) Wrap Up (Appendix 1). Embedded in each section of the SCAG are key adolescent interviewing skill domains that should be addressed by a physician during each phase of the interview. For example, Section 1 includes such items as, ‘Discusses Confidentiality’; Section 3 includes the risk taking domains in an adapted HEADSS format36. HEADSS is a well known mnemonic used to remember the key risk-taking elements of adolescent-hood (see Appendix 1 - Teen Alone Section of SCAG). The SCAG also includes General Ratings, (adolescents prefer this term to Global Ratings), for each section. This allows the adolescent to give an overall score for each section. Further, each section provides room for the adolescent to critique the physician. Clear instructions and examples to facilitate the writing of these examples are given (Appendix 1 – right hand column).
The current design of the SCAG has been achieved through formal research and informal feedback from users and physicians.37 This iterative development has produced a concise and simple guide, which is at a grade 7-8 reading level (12-13 years old).
The study goal and research questions
Study Type: Population cohort intervention
Study Period: 2005-2006
Study tool: The Structured Communication Adolescent Guide (SCAG) (http://services.aamc.org/30/mededportal)
Can untrained adolescents use the SCAG reliability to rate an adolescent-physician interview?
What is the effect of demographic variables (age, sex and school location) on the SCAG score ratings?
How do adolescents score the SCAG when compared with a ‘gold standard’?
What is the adolescent response to using the SCAG?
Adolescents (12-15 years old) from three Junior High Schools located in the Halifax Regional Municipality participated in the study. The schools were situated in rural, suburban and urban regions and were specifically selected to create a broad, inclusive and demographically diverse sample. Specific classes to be included were selected in consultation with the principal.
All students in the selected classes were invited to participate. Participants were included in the study if they returned a consent form, signed by themselves, and a parent or guardian. In the weeks prior to the study, homeroom teachers distributed these consent forms to their classes, together with written information describing the background, method, objectives and relevance of the proposed research. Participation rates were high among those invited to join the study.
Researchers visited each school after having consulted directly with individual classroom teachers to find an appropriate time. Most sessions took place in the students’ homeroom class with the teacher present. Researchers introduced themselves to the students and gave a brief explanation of the task. The students were asked to watch a videotape for 25 minutes which involved a female physician with a female adolescent (SP) aged 14 and her mother (SP). They were asked to imagine that they were the adolescent in the videotape and that they were being asked the questions from the physician. After the adolescents watched the videotape, they were asked to fill in the SCAG as if they were offering feedback to the physician. Therefore, they were putting themselves in the “female SP adolescent’s shoes”, which, we acknowledged, could be a little more difficult for the male students in the audience. Participants were then asked to complete a questionnaire about their experience with the SCAG. The questionnaire consisted of closed- and open-ended questions, so the adolescents could write both what they liked and what they disliked about the SCAG. The questionnaire had been piloted previously with a different school; adolescents at this school had helped construct the questionnaire language to be simple and clear, and framed similarly to other questionnaires they had to complete in school.
Researchers then collected all materials, and the students were led in a classroom discussion that served as a debriefing and an opportunity for the adolescents to ask health related questions. The post-study debriefing sessions lasted between twenty to forty minutes and included discussion of topics ranging from sexual health to mental illness and physical fitness/nutrition.
Physician-Adolescent Clinical Interview Videotape
The videotape used in this study was chosen from a selection of 30 video-tapes by group consensus after viewing all 30 videotapes. This particular videotape was chosen for its balanced of interview presentation, i.e. it was comprehensive but not perfect. The videotape was originally produced for use in a previous research study.25 It featured two SPs (female adolescent age 14 and mother) being interviewed by a female physician. The physician had been told that she would be seeing and interviewing an adolescent patient with her mother for approximately 30 minutes with no physical examination. The interview took place in an office setting, and the appointment was a repeat visit concerning the adolescent’s methylphenidate (Ritalin) prescription. The physician was asked to interview the patient, told that she would be given feedback from her interview, and that this interview would be used for research. Consent was obtained from all participants featured in the videotape.
During the interview, the physician conducted a general introduction and information-gathering phase. She then asked the mother to leave and spoke with the SP adolescent alone (this included discussion of general risk-taking behaviour). The interview was approximately 25 minutes.
The Gold Standard scoring of the videotape was produced by consensus of a pediatrician, an adolescent standardized patient trainer, and a group of standardized patients (n = 8). There was over 90% agreement of scoring of the SCAG among the pediatrician, a standardized patient trainer, and SP adolescents. The remaining 10% of disagreement was resolved with discussion by the standardized patient trainer and the pediatrician. Practically, this involved only three of the SCAG questions.
After the Junior High School adolescents watched the videotape and scored the SCAG, they each completed a questionnaire. Using a 10-point scale (1-Strongly Disagree, 10-Strongly Agree) they rated the following statements:
The guide [SCAG] was easy to use
The language was easy to understand
I would use the guide [SCAG] if it were at my physicians office
I was comfortable with the questions
The guide [SCAG] dealt with important issues
Four open- ended questions asked the adolescent participants to write about their experiences using, and thoughts about, the SCAG.
The questions were as follows:
1. What did you like about the SCAG?
2. What did you not like about the SCAG?
3. Are there any questions you would change/remove from the SCAG?
4. Are there any questions you would add?
Ethical approval for this research was obtained from both the Halifax Regional School Board Research Division and the Dalhousie University Office of Human Research Ethics.
Adolescents’ ability to use the SCAG
The quantitative analysis was designed to assess the first three of our research questions 1) the reliability with which untrained male and female adolescents could use the SCAG, 2) whether the SCAG score differed significantly as a function of adolescents’ age, sex and school location, and 3) the agreement of adolescent participants’ ratings of the physician’s performance with a ‘gold standard’ rating of the same interview.
Descriptive statistics were calculated from the adolescents’ responses to the questionnaire with the five questions rated on a Likert scale (1-Strongly Disagree, 10-Strongly Agree). Sub-group means and standard deviations were also calculated as a function of participants’ age and sex.
The qualitative analysis included an evaluation of the four open-ended questions on the questionnaire that the adolescents completed about the SCAG.
Participation rates were high from all three schools. From a total of 228 eligible students, 183 participated (80.3%). The mean age was 13.4 years (+/- 0.51) years and included male (N = 94) and female (N = 89). Seventeen females and 28 males were unable to participate, either because they were absent, engaged in other activities or had no consent signed consent form.
The mean total SCAG score (Figure 1-A) and the mean score of the General Rating (Figure 1-B), were not significantly different when compared across age or sex of the adolescent participants.
Participants’ age, sex and/or school did not significantly affect the SCAG ratings for each section (Table 1).
The General Ratings and total scores showed overall high reliability (Figure 2) both for male and female adolescent scores. When calculated for each section, the reliability remained moderate to high [Section 1 (‘Getting Started’), 0.44; Section 2 (Gathering Information), 0.61; for Section 3 (‘Teen Alone’), and 0.78 Section 4 (Wrap Up) 0.65]., with the exception of Section 1)
Participants’ mean total score for the interview deviated from the ‘Gold Standard’ score by a value of 0.75. This difference was not significant (P = 0.29). The Gold Standard mean scores for individual sections, General Ratings and total score did not differ significantly from the adolescent mean scores (Figure 3).
Adolescent Written Examples about the physician (Appendix 1, right hand column of SCAG)
Of the 183 SCAG forms completed in the research study, 106 adolescents wrote feedback for the physician in the “comment box” on the right hand side of the SCAG: 47 male and 57 female (106/183 = 58%). A research assistant transcribed all comments. These were then grouped by two of the researchers (KB and SW) to produce themes that typified both the positive and negative comments (Table 2). The adolescents who gave the higher scores of general ratings supported this with positive written examples about the physician, 90% of the time. The adolescents who gave the lower scores of general ratings supported this with constructive comments 82% of the time.
All participants in the study completed the questionnaire (5 Likert style questions) and the mean values of their ratings are presented in Figure 4-A. Ratings were generally high, ranging from the lowest mean score of 6.0 (sd = 2.7) for question 3 (I would use the SCAG if it was in my physician’s office) to 7.8 (sd = 2.2) for question 4 (I was comfortable with the questions). Males tended to give lower ratings than females (Figure 4-B) however, the ratings did not differ significantly across age groups.
Of the 189 participants, 103 (55 %) completed some or all of the four open ended questions on the questionnaire. For Question 1, What did you like about the SCAG?) 65 responded: 32 discussed the ease of using the SCAG and 33 dealt with the important issues covered by the SCAG. For Question 2, What did you not like?, 28 responded: 14 wanted more rating choices and 14 described the SCAG as too long and/or too hard to understand. For Question 3, Are there any questions you would change or remove?, and Question 4, Are there any questions you would add?, 91 adolescents reported that they would not have changed or removed any questions while 85 reported that they would not have added any questions.
The findings of our study support the view that adolescents, aged 12 – 15 years, can easily and reliably complete the SCAG while watching a videotape of a physician-adolescent interview. Further, the scoring was not significantly different when age, sex and school locations were examined. Scoring of the SCAG by the adolescents was not significantly different from that of the ‘Gold Standard’ scores; lastly, participating adolescents found the SCAG usable and relevant.
We were concerned that the SCAG scoring would differ in the males compared to the females, given what we know about the maturity of the sexes. Also, the female patient in the videotape may have hindered the ability of the male adolescents to put themselves in the “female SP adolescent’s shoes”. This may have been one of the reasons that the reliability of Section 1 ( Getting Started) was lower. Given all the results, however, our study supports the use of the SCAG by adolescent males, a finding not previously reported.
Our reliability analysis suggests that adolescents who have not been formally trained can reliably use the SCAG to rate a patient-physician interview on a videotape.
Results from the Gold Standard showed that, on average, adolescents produced an overall SCAG score that was not significantly different from one produced by a group of SP adolescents, a pediatrician and an SP trainer for the same interview.
The questionnaire that the adolescents completed after they had used the SCAG revealed positive experiences. The lowest scored item was that the adolescents would be less comfortable about using the SCAG in their physicians’ office. This may reflect that adolescents are not sure about discussing these types of issues with their physicians. Many studies support this and the fact that the physicians should be asking the adolescents for their opinions and pr providing a tool like the SCAG to the adolescent for anonymous feedback.
In initiating this research, we had reasoned that there would be numerous advantages to having a structured communication guide that could be used by adolescents for assessing the communication skills of medical students, residents and physicians. Medical students in their clerkship at our University are required to have at least one adolescent patient use the SCAG to assess them. Our study lends indirect support to the use of these assessments in the medical students’ evaluations. We suggest that the medical students should be required to have a number of adolescent patients assess their communication skills to ensure maximum reliability.
We also believe that the SCAG may be a useful tool for physicians in practice to use toward improving their adolescent clinical interviewing skills. In this role, the SCAG provides an easily utilized feedback measure that physicians’ could give to adolescent patients they see in their practice. Our study suggests that adolescent ratings using the SCAG could be reliably used to provide formative feedback to physicians.
A number of the adolescents who have been involved with developing the SCAG have asked “Can I get a copy of the SCAG and use this with my physician”? This is an interesting and unexpected finding of our research. The use of the SCAG as an educational tool for the actual adolescent patient was not directly addressed in this study. However, we would like to emphasize the SCAG’s potential in this regard in part because adolescents often do not receive sufficient preventative counselling and screening service37, 38. Adolescent interviewing needs to be addressed through training of both medical students and physicians in practice39 and further, from a patient perspective 40 .
The SCAG is a well studied guide to learning and to implementing a basic adolescent clinical interview; it includes important questions that deal with health-related behaviours that will influence health later in life. Introducing current and future physicians to the SCAG may provide educators with a simple and inexpensive tool that could assist physicians with their vital task of not only getting, but also keeping their adolescent patients healthy.
Evaluating patient-physician communication is a complex and subtle task. Current checklists of observed behaviours are widely used as they are thought to capture essential facets of the patient-physician interaction. Recent research, however, suggests that patient perception of physician communication does not always accord with ratings derived from OSCE-style checklists.35 Hence, in certain instances, it may be more informative for both learners and evaluators to have access to patient (or SP) derived ratings that can serve as an adjunct measure of physician communication skills.
Firstly, this was a school-based study of 12 – 15 year olds used as a convenience sample due to the start and finish of junior high school. It is possible that with a sample having a larger age range the SCAG reliability may revealed more substantial differences. However, this is the middle adolescent period where the risk-taking often starts and this age range is a good target for intervention. Secondly, our study included an evaluation of the SCAG based on participants’ ratings of one videotaped interview featuring a female physician and a female SP adolescent only. This may be one reason that a larger number of female participants provided narrative comments in the “examples section” of the SCAG than did male participants. A third limitation of this study is that although participants were from urban, suburban and rural settings, all were from the Halifax regional municipality. This makes it difficult to determine if the results can be generalized to broader cultural populations, or to adolescents who speak different languages. In summary, the overall results from this study were promising; it is clear, however, that additional work is necessary to address these limitations.
Adolescents aged 12-15, who have not been formally trained in using the SCAG, can easily and reliably rate a videotaped physician-adolescent interview. Age, sex, and school location did not significantly change the adolescents’ ratings, nor did their ratings differ significantly from the “Gold Standard”. Overall adolescents found the SCAG easy to use, and the majority would not have changed or added any questions. That the adolescent participants in this study were untrained in using the SCAG suggests that in real-life settings, such as on the wards, or in out-patient clinics, the SCAG may be useful as a structured source of feedback on adolescent clinical interview skills for learners and physicians alike.