In 2015, some 15,000 nurse practitioners students were prepared to join the ranks of the more than 205,000 practicing Advanced Practice Nurses (APN) in the United States (US) workforce.1 To assure a match between education and clinical competencies, national certification is sought by the graduates and is a state legislated eligibility mandate for licensure to practice as an Advanced Practice Registered Nurse (APRN) in a majority of states. Therefore, graduate nursing faculty are beginning to implement standardize certification predictive testing, exit exam benchmarking, and remediation strategies to facilitate APRN graduate student success on the national certification exams.

The Elsevier HESI™ APRN Exam (APRN Exam) was developed to assess students' practice knowledge within specific role content areas such as the Primary Care Family or Adult-Gerontology Nurse Practitioner. The APRN Exam’s individual student assessment report can be used to direct remediation to strengthen student knowledge and clinical acumen. The APRN Exam’s cohort assessment report identifies curricula strengths/weaknesses and can provide evidence of the graduate program’s outcomes. Many studies support the use of Elsevier HESI RN® standardized exams for undergraduate end of program assessment. The HESI Exit Exam has been found to aid knowledge development and ensure competence of students seeking licensure in vocational nursing, associate degree nursing, and baccalaureate degree nursing programs and to demonstrate organizational and curriculum effectiveness.2-9 There is limited guidance for faculty in how to implement standardized predictive testing within graduate nurse practitioner programs. 

A review of the literature was undertaken using the search terms standardized testing, nursing, HESI, advanced practice nurse, nurse practitioner for 6 databases (CINAHL Complete, EBSCO, PubMed, Medline, Mosby’s Nursing Consult, and Nursing Reference Center). Resultant retrievals were few and none were APN specific (e.g., CINHAL Complete retrieved 1,778 hits for standardized testing but when all search terms were included no articles were found). So the grey literature was searched. Google Scholar found 2 documents; 1 was an Elsevier product brochure (2011)10 and the other a National Organization of Nurse Practitioner Faculties (NONPF) conference podium presentation abstract and participant handout (2011, April).11 The Sigma Theta Tau International Virginia Henderson Global Nursing e-Repository search revealed 3 pertinent conference abstracts. Olsen, Stewart, Merriman, and Nibert (2006) presented the initial development and validation of the APRN Exam, student individual reports, and cohort summary reports.12 The study was a multi-site, convenience study (N=49); the predictive validity was 100% for students achieving a HESI score of 800 or higher on the Family APRN Exam. Binder and colleagues presented how a university system with multiple campuses implemented the computerized APRN Exams to predicted APN success on certification exams from 2005-2007.13 The convenience sample (N=170) was found to predict 100% certification success for both Family and Adult-Gerontology APN students whose HESI scores were 800 or higher. A three-year validation project (N=118) by Willson found that regardless of credentialing body (American Nurses Credentialing Center [ANCC] or American Academy of Nurse Practitioner Certification Program [AANPCP]), both Adult-Gerontology and Family Nurse Practitioners APRN Exams were 100% predictive when HESI scores were 800 or higher.14Foundational content, construct, and predictive validity testing for the APRN Exam has been presented to faculty in several venues. Elsevier has recommended APRN HESI Score benchmarks of 800 and higher with a minimal acceptable score of 750.10The aim of this study was to determine the APRN Exams predictive validity by scoring categories in a national US sample. Students APRN certification predictive rates by score categories, faculty’s implementation strategies, and policy development was investigated.

Methods

            An ex post facto design was used to compare first-time certification passage and APRN Exam HESI scores of APN students. After obtaining Elsevier, Inc. and institutional review board approval, an electronic invitation for participation was sent to directors of all graduate nursing schools that administered Elsevier HESITM Advanced Practice Registered Nurse (APRN) Exam for Family and/or Adult-Gerontology APN students during the 2010-2011 academic year (see Figure 1). Data were collected regarding students’ APRN national certification first-time examination outcomes and the schools’ implementation strategies and policies. Datasets were entered into IBM SPSS version 18 and analyzed using descriptive statistics. 

Instruments

Elsevier HESI APRN Exam (APRN Exam). Elsevier, Inc. as the test producer delivered both the APRN Exams and the Certification Outcomes Questionnaire to ensure exam security and that customer (school/student) data were de-identified in a password protected electronic format. The APRN Family or Adult-Gerontology Exams consisted of 110 multiple choice and multiple answer items. Ten of the testing items were pilots and did not count toward the students’ grades. The exam was taken electronically in a proctored environment, and at the conclusion of the testing the student had the option to review missed items and answer rationales. Individualized student score reports were printed, and faculty cohort summary reports were available for faculty. The APRN Family and Adult-Gerontology Exams were designed to reflect the AANPCP and ANCC testing blueprints for the APN specialty role. The APRN Exam reliabilities ranged from .90 - .92 for the testing pool. Item uses ranged from 466 to 2394 per item.

Certification Outcomes Questionnaire. The Certification Outcomes Questionnaire was unique for each school/program director as it listed student cohorts, APRN HESI scores, and requested first-time certification outcomes (Pass, Failed, Unknown). Directors indicated student outcomes, removed student identifiers, and returned the questionnaire to Elsevier, Inc. where a de-identified dataset was constructed. The dataset included APRN Program type, APRN exam type, APRN HESI score, and Certification outcome.

APRN Exam Implementation Questionnaire. The APRN Exam Implementation Questionnaire was an investigator-designed 13-item multiple choice with open-ended comment fields questionnaire delivered by a private link to the SurveyMonkey® platform. The directors and chairs were asked if the exam counted toward the students’ course grades, if there was a policy regarding the APRN exam and if so, what consequences occurred it the student did not meet the benchmark score. Additionally, nurse educators were asked about remediation strategies, when testing occurred, and how testing data were used within their programs. Lastly, faculty was asked about their satisfaction with the Elsevier’s HESITM APRN Family or Adult-Gerontology Exams and reports.  

Results

Sample. Invitations for participation were sent to 35 programs in one electronic distribution. Non-responders received one reminder notification in two-weeks; 9 programs joined the study for a 26% response rate. The program types were 3 Family Nurse Practitioner (FNP) Programs, 3 Adult Nurse Practitioner (ANP) Programs, and 3 schools who offered both curriculums. This sample represented 141 students; 96 attending FNP programs and 45 attending ANP programs. Of the 9 schools, only one graduate program had an end of program testing policy. Three schools counted the HESI APRN exams for 10% of grade, and the benchmark used by the 3 schools was 800.

Predictive validity. The APRN HESI scores ranged from 484 to 1039 points, with a mean of 796 (SD = 113.39) and a mode and median of 805. The number and percent of student testers are reported in Table 1 by scoring category and APRN HESI score. Fifty-three percent of the students scored at 800 or higher on the exam. The predictive accuracy for each scoring category is presented in Table 2.  All students (n=113; 80.1%) scoring in categories A through F (700 – 900+) passed their certification examinations.

Evaluative usefulness. Two Likert scale response questions investigated the faculty’s satisfaction with APRN Exams evaluative usefulness. The first item asked about faculty satisfaction with the test as an indicator of student competencies and the second inquired about usefulness as an indicator for curricular outcome achievement. From a 10-point scale where 1 equaled not satisfied and 10 was very satisfied, the mean score for student competency indicator was 8.13 (most satisfied) and satisfaction as curricular indicator was 8.25 (most satisfied) for the 8 programs who gave the APRN Exam at the end of the program. One school gave the exam at the beginning of the senior year and ranked the usefulness at measuring student competencies and curricular outcomes as a 3 (somewhat satisfied). Results indicate that the timeframe of assessment may affect the exam’s evaluative usefulness.

Discussion

Competency Outcomes

Certification bodies determine testing content based on periodic workforce surveys and publish certification examination test blueprints.  For example, the AANPCP’s testing domains in adult-gerontology and family nurse practitioner examinations includes Assessment, Diagnosis, Plan, and Evaluation15and those for ANCC are Foundations for Advanced Practice, Professional Practice, and Independent Practice.16 Elsevier matches certification exam blueprints and reports outcomes of the APRN Exams by AANPCP and ANCC categories.  Student scoring reports identify both accreditation bodies’ test blueprint categories and provide the student HESI score for each category as well as the national normed score for each category.  Therefore, individual student scores and cohort scores can be tracked and trended over time by any or all of the certification body testing domain categories. In this study, faculty reported that the individual student reports helped identify the student’s learning needs so that remediation strategies could be implemented by a personalized didactic and/or clinical plan of action.

Course and Program Evaluation

            Standardized testing can provide outcome measures for curriculum evaluation, program accreditation, and program policy development. The APRN certification exams success is one metric used to measure APN knowledge and competencies. When a standardized exam is given at the end of a course (e.g., Advanced Health Assessment, Advanced Pathophysiology, or Advanced Pharmocotherapeutics) as the course’s summative exam, the course content, concepts, and teaching/learning strategies can be evaluated as compared by course cohort and national cohort scores. Likewise, when standardized testing occurs at the end of the program of study, such as a Primary Care Family Nurse Practitioner Program testing with the Family APRN Exam, the mean cohort scores can help identify program curricula strengths and weakness. The evidence acquired directs faculty’s evaluation and change plans.

Program evaluation is usually aligned with the institution’s effectiveness plan as well as the accreditation process. Certification examination pass rate is one of the important outcome measure included in the accreditation standards for programs. For 2014, ANCC reported initial certification examination pass rates for primary care Family Nurse Practitioner (FNP) test takers as 74% (N=5,106) and 81% (N=1,521) for Adult-Gerontology Nurse Practitioner (A-GNP) testers.17 For the same year, AANPCP reported initial certification pass rates of 88% (N=624) for FNP testers and 86% (N=622) for A-GNP test takers (note: the numbers of test takers reported by AANPCP includes persons who sat for the examination for the first time, as re-takers, and those who recertified by exam instead of by continuing education activities.18 Graduates who successfully complete the certification examination on first attempt provide valuable assessment data validating the program effectiveness. Policy decisions faculty make about pre-assessment testing often reflect the school of nursing’s desire to meet program evaluation goals, such as “100% of FNP graduates will pass initial certification exams” that was used at these authors’ schools. While the graduate faculty members were aware that 100% pass rate was a lofty goal, the faculty desired the objective and their contingency plan if unmet was to implement teaching and learning strategies to accomplish it. Therefore, faculty looked for strategies that push their school pass rates higher than what was occurring nationally as reported by certifying bodies. However, the evidence guiding graduate faculty decisions was based almost entirely on undergraduate education experiences, which are highlighted in Table 3.

Conclusion

            A national multi-site US sample of graduate programs has found that electronic standardized end of program assessments are highly accurate at predicting APRN certification success for those students whose APRN HESI scores were 700 or higher. These findings are consistent with preliminary benchmark setting studies for APRN HESI minimum scores of 750 or higher. Faculty’s implementation strategies, policy development, and methods for curricular evaluation were discussed and are limited by the reports of only 9 schools. Most schools administered the APRN Exams at the end of the program of study. Minimal consequences (10% of the course grade) were applied in a small subset of the schools. Graduate faculty have a limited track record of using predictive standardize exams, even though at times the same faculty have used end of program predictive exams in undergraduate education. Therefore, graduate faculty must continue to look to undergraduate education experience with standardized testing for development for guiding curriculum, setting the direction for future programs, benchmarking, admission standards, and policies. This report is a building block for the educational science on APRN predictive testing and should be strengthened by further studies.