In assessment, the fundamental property of any testing method is that ““it measures what it purports to measure””. ¹⁰ This, in essence, is an assessment’s validity. While an outwardly simple concept, validity testing often requires the availability of other frameworks to which the results of the index assessment can be compared.¹¹ A test then, may have multiple aspects of validity.^{5, 11} These aspects may be compiled to establish the overall validity of a particular assessment method. To rationalise its multiple facets, several standards have been developed in the educational literature to appraise the validity of an assessment instrument. These standards include face, content, construct, and criterion validity.

Face Validity

Content Validity

Construct Validity

Criterion Validity

Reliability relates to the precision, stability or reproducibility of an assessment tools results. In basic mathematical terms, reliability is estimated as:

R_x = V_T/V_x

Where: R_x is the reliability in the observed (test) score, X; V_t and V_x are the variability in ‘true’ (i.e., candidate’s innate performance) and measured test scores respectively.

Simply stated, reliability is a term that covers the dependability of an assessment and measures the extent to which a test will yield the same result after multiple administrations under the same conditions.¹⁰ Reliability is recorded as a coefficient on a scale from 0 to 1. A test with a reliability coefficient of 0 is completely unreliable. That is, the variability in test results are independent of candidate ability. A test with a coefficient of 1 indicates complete reliability and is rarely achieved. There is general agreement that if important decisions are going to based on the results of a test, a reliability of  0.8 is required. ⁵

In general, the reliability of an assessment is easier to determine than validity. Like validity, there are a number of methods to establish a test’s reliability. Important methods include internal consistency, test-retest, equivalent forms, and inter-rater reliability.

Internal Consistency Reliability

Test-Retest Reliability

Equivalent Forms Reliability

Inter-Rater Reliability

Any assessment mechanism must be accountable to all ‘stakeholders’ involved. This is a fundamental principle from which the other characteristics of the ideal assessment tool should arise. In academic medicine, these stakeholders include students, clinical educators, the program and institution, licensing bodies and ultimately the community that the clinician will serve. To facilitate accountability an assessment must be defensible and able to provide a logical analysis or explanation of results. ¹²

Clinical medicine is practiced in a diverse and sometimes unpredictable environment. Therefore, the chosen assessment method must be flexible and allow the examiner to evaluate the complete clinical spectrum of the content domain in question multiple times and in multiple settings (e.g., elective and emergency surgeries).

To be effective overall, an assessment will evaluate all pertinent objectives and document corresponding examinee performance for the course it was designed to evaluate. The CanMeds competency framework, which defines seven essential physician competencies (Medical Expert, Professional, Communicator, Collaborator, Manager, Health Advocate, and Scholar), illustrates the robustness of clinical practice.¹³ With this view of practice, the clinical educator can better understand the importance of comprehensive mechanisms to assess trainees.

To facilitate acceptance by all stakeholders, an assessment should be portable, cost-effective, practical, and limit physical and human demands. This is important, as an assessment that taxes limited resources, is unlikely to find mainstream acceptance. However, under some circumstances, these considerations may be tempered. For example, for licensure or other high-stakes examinations, a more labour intensive and expensive assessment may be used if it is proven to be superior to other available tests. Examinations such as the Objective Structured Clinical Examination or Objective Structured Assessment of Technical Skills (discussed later) are examples of resource intensive but valuable assessment tools.

To maximise its function, assessment should be administered as close to the target behavior as possible. Undue delay allows for recall of target events to degrade and thus increases the subjectivity of assessment. Also, the results of the assessment should be communicated to the examinee (and other relevant parties) promptly. Failure to do so deprives stakeholders of the assessments full utility (e.g., feedback or curriculum planning functions). Indeed, if documentation is delayed, assessment is less effective as a learning tool, more subject to bias, and less defensible. ¹⁴

To be effective, the importance of the assessment must be apparent to all involved stakeholders. The results of assessment, favorable or not, must be used to facilitate learning and influence promotion and curriculum planning decisions. An assessment that is viewed as irrelevant cannot fulfill these functions because its results appear meaningless and unusable.

To date, a range of assessment techniques has been described and utilised in all areas of medical education. Although too numerous to describe, each method has its own inherent advantages and disadvantages. When choosing a method, it is important that the assessment technique be closely related to what one is trying to examine. ⁴ This concept is reflected in Miller’s triangle model which attempts to stage clinical competence. ¹⁵ In this model, the cognitive and behavioral progression a learner makes from acquiring knowledge to performing a task is illustrated in four stages. These are: knows, knows how, shows how, and does (Figure 1).

For example, if the aim is to examine a candidate’s factual recall (““knows””), a multiple choice or extended matching item examination may be sufficient. If a candidates thought process is the target (““knows how””), an essay format or oral examination may be useful by allowing a free and extended arena to formulate a response.

Figure 2

Figure 1. Millers Triangle

In clinical medicine it is important to distinguish between what a candidate knows and what they can do (““shows how””). Here, the clinical and practical assessment techniques are important. These techniques importance have lead to more objective approaches to clinical assessment over the past 30 years. The Objective Structured Clinical Examination (OSCE) and more recently the Objective Structured Assessment of Technical Skill (OSATS) and Patient Assessment and Management Examination (PAME) are well known examples of these. ^{9, 16, 17}

Miller’s triangle assumes ^{that competence will predict actual clinical performance (““does””). However, this may not be the case as many other factors can influence clinical performance. To address this, the ““Cambridge Model”” (Figure 2) expands Miller’s triangle to illustrate individual and system related influences that effect performance. 18 This model distinguishes competence (what a candidate demonstrates during the examination) and performance (what the candidate demonstrates in real practice). With this model, the authors illustrate the need, when appropriate, to assess true clinical performance in addition to ““in vitro”” assessments of knowledge and practical skill.}

Figure 3

Figure 2. The Cambridge Model for delineating performance and competence.

Formative assessment involves gathering findings from a variety of assessment sources. These findings are then used to chart a student’s progress through a particular course of learning. Importantly, formative assessment uses information to ‘feedback’ into the learning and teaching process which can be used for either student or program assessment. ^{5, 20} In student assessment, a formative assessment is intended to give ongoing constructive feedback on a student’s strengths and weaknesses during a course. This feedback is student-centered and does not focus on the student’s ranking within a particular group. In program assessment, this assessment aims to improve the quality of the program. In neither student nor program assessment is formative assessment used to make pass or fail decisions. ⁵

In contrast to formative assessment, summative assessment is designed to accumulate information from all relevant sources and to determine whether course objectives have been adequately met. Summative recommendations usually occur at the end of a course and are used to make pass/fail/rank decisions. The intention here is to determine what has been learnt.

For more information on formative and summative assessment please see Wanzel et al. ⁵

Norm and criterion referencing are two common methods of relating a student’s raw performance against a standard so that comparisons or rankings can be drawn. Norm referencing is the more conventional method and is used to describe a candidate’s performance in terms of their position in a group. Results are usually reported as a percentage of correct responses where the number of students to pass or be given a particular grade has been predetermined. Students often refer to this as the grade ‘curve’ for the class. Conversely, criterion referencing has particular importance in professional education where there is more concern that the student attains a minimal level of competence rather than focusing on their ranking within a peer group. ““Standards of performance are set using minimal levels of competence before the test is applied. Following the test there is no attempt to alter the percentage of pass or fail. The standards of performance are based upon uncompromising pre-set objectives.”” ²¹

Both forms of judgment have merit in different circumstances. For example, norm referencing is useful when determining which candidates from a pool should be selected for a limited number of positions in a medical school. On the other hand, criterion referencing would be more appropriate in determining who graduates from medical school because a significant percentage or all of the candidates of that pool may be particularly skilled.

For more information on norm and criterion referencing please see Turnbull.²¹

This method is the most widely utilised and accepted technique in evaluating medical and surgical trainees.⁵ The strength of this approach draws from an underlying belief that experts (i.e., physicians) are better able than non-experts to discriminate between the subtleties and intricacies of their field. This is supported by data which show that global rating scales (which require the rater to understand of the content domain) are more accurate then checklists for assessing a technical skill when administered by expert. ²²

Self-assessment is theoretically appealing because it allows the learners to take ownership of their own educational process. In the medical community, continuing education is driven by the physician’s assessment of their learning needs and is considered a professional requirement. Despite its appeal, however, results of self-assessment studies in higher education have been mixed. Several meta-analyses have concluded that students are poor to moderate judges of their own performance. ^{23, 24} Others, especially with respect to technical domains, show that self-assessment is quite reliable. ²⁵

When reviewing the self-assessment literature, a number of factors must be considered. First, it is important to remember that the ability to assess and self-assess is not an innate skill. Although it would be convenient, it is unlikely that individuals are able to efficaciously use assessment methods without prior instruction in their use. In addition to this ‘calibration,’ the assessment method used must be rigorously validated. Only with this knowledge can the reader draw meaningful conclusions about the utility of self-assessment in higher education.

Peer assessment has recently garnered attention as an assessment tool. While several exist, a useful definition of peer assessment is ‘assessment of the work of others by people of equal status and power’. ²⁶ While the potential development of learner maturity, critical skills and discernment are evident, there is some reluctance to initiate peer assessment as an assessment tool. Reasons for this reluctance include: learners’ believing they do not have the necessary skills to assess others; tradition’s dictating that the educator evaluates learners; interpersonal relationships’ interfering with assessment; and a perceived unreliability of the results. ²⁷ However, investigation from Belfast indicates that learners are able to make rational judgments about the work of their peers with correlation coefficients between peer and expert assessment being 0.89. ²⁸ These results have been replicated in undergraduate as well as surgical and medical training. This indicates that peer assessment may have a significant role in assessment. ^29-32

Despite its intrinsic appeal, there are limitations to using content experts to examine. These include added expense, scheduling conflicts, and retention of experienced examiners due to changing interest and burnout. To address this, non-physicians have been increasingly utilised in assessment. As outlined by Wanzel, non-physicians have been utilised in both ward and OSCE settings in a cost effective and reliable manner after sufficient training.⁵ However, these settings rely predominantly on checklist ratings that may be inferior to global rating scales in the hands of experts. ²²