The study was conducted at one US medical school with class sizes of 92 to 104. Three consecutive classes of students were tested for their clinical skills using the same four-station OSCE at the end of 18 months of a required longitudinal Doctoring course. The primary goal of the 180-hour Doctoring course was to introduce basic interviewing, physical examination, communication, and clinical reasoning skills to prepare students for third year clerkships. The course started in the fall of the first year and continued to the spring of the second year, and took place concurrently with other required basic science courses including Anatomy, Physiology, Pharmacology, and Pathology. The course comprised eight consecutive organ-system-based clinical cases (matched to teaching in the basic science courses) with opportunities for structured observed interview and physical examination of standardized and real patients in weekly two-hour problem-based learning (PBL) sessions. Students were taught in groups of four to six by a pair of faculty, one a physician, and the other a related health professional (e.g. a psychologist, nurse, or social worker). Small group teaching was standardized using a structured schedule with learning objectives, SP cases, and homework assignments, supported by intensive faculty development, which occurred monthly. PBL sessions were videotaped and feedback on interviewing, physical examination skills, communication, and group participation was given to students. Small group teaching was supplemented by teaching of content themes relevant to the cases, weekly half day preceptorships with practicing community physicians during the second year, and required readings from textbooks. Student evaluations consisted of: individual assessments by PBL instructors twice a year, , assessments by community preceptors, a written test, and the final OSCE given in the second year. Physician PBL instructors had at least 32 hours of face-to-face teaching contact with their small groups and evaluated students weekly on their homework and class performance as well as meeting individually with each of their students once every 2 to 3 months.

The OSCE was administered during the Winter Quarter of the second year to the entire class over a three-week period on eight half days at the medical school’s Clinical Skills Training Center with a level of test security equivalent to other high stakes clinical examinations offered at the school.

Student participants were second year medical students. Faculty raters were instructors in the PBL groups for the students. SPs were trained actors with prior experience as raters in high stakes clinical practice examinations.

The goal of the OSCE was to determine that students had basic interviewing, physical examination and clinical reasoning skills appropriate to the course goals and content. Four cases were developed and pilot-tested in the year prior to the study to address learning in the skill domains and organ systems taught. Each station was 25 minutes long, with a 10-minute inter-station exercise for writing up the case or to answer case-specific questions. The cases were: a 45-year old businessman presenting with atypical chest pain with a differential diagnosis of ischemic heart disease/angina, gastroesophageal reflux and pleurisy; a 60-year old woman with transient neurological symptoms and vascular disease; a 30-year old man or woman with acute back pain secondary to muscle strain from work; and a 23-year old male college student with acute right lower quadrant abdominal pain with a differential diagnosis of appendicitis, urinary infection, kidney stone or testicular torsion.

During the OSCE, faculty observed the encounters live from a television monitor outside the room. They were provided with written instructions for the modified mini-CEX to rate each skill domain based on their impression of skill appropriate for a second year medical student and to provide written narratives to support their ratings. All faculty were introduced to the modified mini-CEX during the Doctoring course, and were trained in the use of anchors for the rating scales within the mini-CEX. We chose to use the mini-CEX for the OSCE because of faculty familiarity with the tool and for consistency with clerkships which also use the mini-CEX for student assessment.

SP ratings: History taking and physical examination skills were assessed using case-specific checklists. Communications skills were measured using the same seven-item, Likert type five-point, rating scale (1=marginal/low to 5=outstanding), and the PPI¹⁵ for all four cases (see table 1 for individual PPI items). The maximum score for the PPI for each case was 35 points. Overall patient satisfaction (OS) was measured using a single-item, 5 point scale with a range from 1= highly dissatisfied to 5=highly satisfied.

Faculty ratings: All faculty raters completed the mini-CEX^3,4,23 once on each student observed at one station. The skills rated were history taking, physical examination, communication, humanism and overall clinical competence (see table 1 for rating domains), with a rating scale from 1=unsatisfactory to 5=exceptional.

Internal consistency reliability: The PPI was the common communication skills measure used by different SPs across the four OSCE cases and its internal consistency reliability was assessed. The entire modified mini-CEX (5 skill domains) was assessed for its internal consistency reliability (table 1) among the different faculty raters.

Comparison of individual SP and faculty skill measures: Four different clinical skills were compared for SP and faculty ratings. The SP history-taking score or Hx was the sum of all history-taking items (binary scoring yes/no or 1/0 for each item) for each case corrected to a percentage score. The matching faculty history-taking score was the single score for question one (CEX1) of the mini-CEX. The SP physical examination (PE) score or Px was a sum of the individual behaviors (binary scoring yes/no or 1/0 for each item) for that station corrected to 90% plus the Fundamentals of Physical Examination Behaviors (FPEB) score corrected to 10% for a total of 100%. The FPEB consisted of 2 questions (binary yes/no or 1/0 for each item) asking if students explained the examination and if they respected the modesty of the patient. The matching faculty PE score was the score for question two (CEX2) on the mini-CEX. The SP ‘humanistic quality/communication score’ was the sum of the PPI score, with points totaled (maximum score for summed PPI =35). The equivalent faculty rating was the score for question four (CEX4) on the mini-CEX. The SP overall satisfaction (OS) item for each station (‘please rate the overall performance of the student’) was matched to the score for question five on the modified mini-CEX asking about the overall performance of the student (CEX5). Question three on the modified mini-CEX (CEX3), asking about the clinical reasoning skill of students, was not matched to any SP rating, as SPs were not asked to rate students’ clinical reasoning skills.

Statistical analysis was performed using the software package JMP version 7.0.1 (SAS Institute Inc., Cary NC). Because analysis by student class showed no differences in means and SD among the 3 classes, the data was analyzed in aggregate. The means and standard deviations for the modified mini-CEX and PPI were calculated. Multivariate analysis was used to calculate Cronbach’s α to establish internal consistency reliability for individual items and combined items of the modified mini-CEX (faculty raters) and the PPI (SP raters).

Pearson r correlations between each mini-CEX item (1, 2, 4, and 5) and the corresponding SP measure (described above) were calculated. CEX 1, 2, 4 and 5 (using the 5-point Likert scale) were compared with each relevant variable: Hx% with CEX1 , Px 90%+ FPEB10% with CEX2 , PPI total score with CEX4 , and OS with CEX5 .

Subgroup analysis was performed to examine the correlations using Spearman ρ, by student gender, student class, faculty gender, and clinical case.

The OSCE was conducted in the winter of 2005, 2006 and 2007. Each of the three classes (N=303 total) was similar in demographics. Mean age was 23 years, 50% were male, and distribution of ethnicity was as follows: 40-50% white, 40% Asian, 10% Hispanic, 1% black, and 10% did not self-categorize. The majority of students were science majors and residents of California.

There were 34 physician faculty raters (16 male) from primary care settings of Family Medicine (n=16), Internal Medicine (n=15) Pediatrics (n= 2), and Psychiatry (n=1). Faculty ranks ranged from Assistant to Full Professor. Individual faculty experience teaching in the Doctoring course ranged from 3 to 10 years. Each faculty rated 3 to 17 students. The majority of faculty (25 of 34) taught students during all 3 years of the study.

Twenty-four SPs performed in the three OSCEs over 3 years and each performed in the same case. There were two SPs for each clinical case who were trained concurrently for 4 hours each, using videotapes of student performance as “““gold standards””“ for rating. Returning SPs were retrained each year for 2 hours with review of the case and rating standards.

Mean (SD) scores out of 5 for modified mini-CEX clinical skill domains assessed by faculty ranged from the lowest mean score of 3.04 (± 0.76) for physical examination, to the highest mean score of 3.52 (± 0.58), for humanistic qualities (Table 1), equivalent to percentage scores of 60.80% to 70.40% respectively. Mean (SD) SP percentage scores ranged from a low of 52.90% (± 16.99%) for physical examination, to a high of 74.85% (± 17.72%) for history taking. The mean score for the PPI summed total was 25.45 (± 3.02), out of a maximum of 35 points and the mean OS score from SPs was 3.69 (± 0.58), out of a maximum of 5 points (figure 1).

Mean scores for each case and by skill domain assessed by SPs did not improve across the three years of the study (data not shown), suggesting good test security across the years of OSCE administration.

The overall Cronbach’s α for the PPI was 0.86. The overall Cronbach’s α for the entire mini-CEX was .89 (table 1).

Both faculty and SPs used the full range of the individual scales (see figure 1 for distribution of scores for measures) with scores demonstrating a normal distribution. The Pearson correlation coefficient r between the modified mini-CEX and the equivalent SP measures were 0.214 for CEX1 and Hx percentage score, 0.288 for CEX2 and Px (90% physical examination + 10%FPEB) score, 0.213 for CEX4 and the PPI summed total score, and 0.245 for CEX5 and overall satisfaction SP score (Figure 1).

Subgroup analysis (table 2) showed that the Spearman ρ was lower in all four modified mini-CEX domains for female compared with male students. The range was 0.10 to 0.18 for females and 0.23 to 0.33 for males. There were otherwise no consistent patterns of correlation by case, student class, or faculty gender (Table 2). However, one of the four cases (abdominal pain, case 3) showed a higher range of variability in concordance (Spearman ρ -0.032 to +0.473) than the other 3 cases (table 2) which may be due to the larger number of SPs used for this compared with the other three cases. When data was examined by individual faculty, no single faculty demonstrated a tendency to score in a narrow low or high range (data not shown).

Figure 1

Table 1: Modified Mini-CEX and PPI Scores and Variances - University of California, Irvine, School of Medicine 2005-2007

Figure 2

Table 2: Correlations (Spearman ρ) Between Faculty and SP Measures by Subgroup Analysis -University of California, Irvine School of Medicine 2005-2007

Figure 3

Figure 1: Correlations (Pearson ) and Variances of the Modified Mini-CEX and Matched SP Measures University of California, Irvine School of Medicine

Note - Number for each measure is lower than 303 due to some missing data