Advanced Practice Registered Nurse (APRN) education is expanding worldwide (i.e., Australia, Canada, Germany, Hong Kong, Mainland China, United Kingdom, and United States [US] of America). In the 2016-17 academic year, an estimated 26,000 nurse practitioners completed their academic programs and joined the ranks of an estimated 248,000 practicing APRNs.¹ Of the APRNs in practice, approximately 49.9% hold hospital privileges and 95.8% prescribe medications. To ensure that APRN graduates are prepared to provide these services and to meet state mandated regulations, certification is required. Schools that provide APRN programs are also evaluated on the ability of their graduates to pass the certification exam and become licensed as APRNs in their respective states. As a result, APRN programs have increased the utilization of standardized testing as an educational strategy and program outcome measure. Faculty members have begun to implement standardized testing to predict graduate success on the certification exam. These exams must be validated on a regular schedule to ensure that they are still predictive of success on the certification exam with varying groups of students and any revisions in the certification exam.

The two APRN certification bodies are the American Academy of Nurse Practitioners Certification Board (AANPCB) and the American Nurses Credentialing Center (ANCC).  The initial exam pass rates for AANPCB in 2017 were: Family NP-80%; and Adult-Gerontology NP- 82%.² A total of 18,987 graduates took the AANPCB exam. The initial exam pass rates for 2017 for the American Nurses Credentialing Center (ANCC) were: Family NP-83%; Adult-Gerontology Primary Care NP- 77% of the 6,487 exam takers for these specialties.³ Graduate APRN faculty are using standardized testing to: 1) assess students’ potential for success on certification examinations, 2) to identify concepts needing additional remediation, and 3) to identify areas of strength and weakness within the APRN curriculum. For the purposes of this study, the Elsevier Health Education Systems, Inc. (HESI™) exams were used.⁴ The HESI ™ APRN Exit Exams consist of 100 scored items with content matched to ANCC and AANPCB’s published certification examination blueprints. HESI APRN Exit Exam scoring reports are categorized according to ANCC and AANPCB domains of practice and the National Organization of Nurse Practitioner Faculty (NONPF) competencies.⁵ For colleges and universities providing nursing curricula leading to APRN licensure, HESI™ provides standardized exams to assess application of course content and an exit exam specific to the student’s specialty area.

Background

Four small studies have been completed using the Elsevier HESI™ APRN Exit Exam, the original development study and three studies by APRN faculty. In 2006, Olsen, Stewart, Merriman, and Nibert conducted a multi-site study with a convenience sample of 49 students who took the Family Exit Exam which resulted in a recommended benchmark score of 850.⁶ Binder and colleagues evaluated the predictive validity of the exam with 170 students from three campuses of the same university. The predictive validity for students scoring 800 or greater on the Family or Adult-Gerontology exams was 100%.⁷ Willson completed a three-year validation study with 118 students taking the Family and Adult-Gerontology Exams. This researcher found that the predictive validity for both exams when students scored 800 or above was 100% regardless of the credentialing agency. In 2015, Willson and Goodman surveyed 35 programs with 9 responding, three Family, three Adult-Gerontology, and three that offered both curricula.⁵ One hundred forty-one students were represented including 96 Family and 45 Adult-Gerontology NP students. Scores varied from 484 to 1039 and 53% of the students scored 800 or higher. All students (n=113) who scored 700 or greater passed their certification exam. Two questions with 1(low satisfaction) -10 (high satisfaction) Likert scale responses were asked of program directors. Satisfaction with the exit exam as a measure of student competency resulted in a score of 8.13 on the 10-point scale. Satisfaction with the exam as a curriculum evaluation tool resulted in a score of 8.25. The authors indicated that the timing of the exam may have impacted its predictive usefulness. For example, if testing occurred at the beginning of the student’s final semester and not within 6-8 weeks prior to finishing the program of study, the scores would not be reflective of knowledge gained by the student’s content remediation or clinical experiences prior to testing.

Conceptual Framework

A review of literature specific to testing theories produced four reports ^8-11 which addressed nurse testing theories. Three specific testing theories were identified: 1) the Classical Test Theory by Crocker and Algina⁸, 2) the Critical Thinking Theory by Paul¹¹ and 3) the Reconceptualization of Vygotsky’s Theory of Goal Attainment by Eun et al.⁹ Two of the three theories, the Classical Test Theory⁸ and the Critical Thinking Theory,11 were specifically used in undergraduate test item writing for reliability, validity, standard error, criterion, and norm-criterion referenced measurements and did not address standardized testing nor advanced practice. However, the Reconceptualization of Vygotsky’s Theory of Goal Attainment by Eun et al.⁹ was used in graduate APRN nursing education related to standardized testing policy,¹⁰ thus this theory was chosen to underpin the conceptual framework of standardized testing research for this study.

In 2008, Eun et al. Reconceptualized Vygotsky’s Theory of Goal Attainment⁹, the authors specifically considered the concepts of first and second voice. Furthermore, they proposed the competent person as the experienced, first voice and the less competent person as the inexperienced, second voice. Eun and colleagues suggested that a hidden, inaudible voice exists in the process of goal attainment, thus introducing an additional new concept in their reconceptualization of the theory as a third voice. This third voice makes evident those elements that influence development, mediating the perceptions and interpretations of the first voice and second voice. ^{9, p.136} First voice is conceptualized as the faculty, second voice as the student, and third voice as standardized testing, a silent, inaudible participant that mediates interpretations of events within the zone of proximal development (ZPD) as test results. In fact, Barton et al.¹⁰ speak directly to this conceptual framework by Eun et al.⁹ underpinning standardized testing in APRN nursing education.  The standardized testing results are considered the objective, silent third voice in the journey towards goal attainment.¹⁰

Design and Methods

An ex post facto descriptive comparative design was used to assess the exit exam and the first time certification pass rates were for each of the scoring intervals on the exit exams. Electronic surveys were mailed to deans and program directors of 114 graduate nursing schools that utilized the HESI™ APRN Exam for Family and/or Adult-Gerontology for students during the 2013 - 2015 academic years using the cloud-based SurveyMonkey® software.¹² Descriptive statistics and certification predictive pass rates were calculated using IBM SPSS Statistics - 25.¹³

Instruments

The study had two phases: Phase I and Phase II. In Phase I, the HESI APRN Exit Exam Practices/Strategies Questionnaire, was sent to deans/directors via a private internet SurveyMonkey link. The request for participation was repeated again in two weeks for non-responders. When Phase I survey responses were received, the Phase II survey, Certification Outcome Sheet by Student HESI Scores was sent using a password protected methodology. The surveys explored the programs’ testing policies and practices as well as reports of students’ APRN national certification outcomes.

Elsevier HESI APRN Exam

Elsevier HESI APRN Exam (APRN Exam). Elsevier, Inc. as the test producer delivered both the APRN Exams and the Certification Outcomes Questionnaire to ensure exam security and that customer (school/student) data were de-identified in a password protected electronic format. The APRN Family or Adult-Gerontology Exams are both made up of 110 multiple choice and multiple answer items.⁴ Ten of the testing items were pilots and did not count toward the students’ scores. The exam was taken electronically in a proctored environment, and at the conclusion of the testing the student had the option to review missed items and answer rationales. Individualized student score reports were printed, and faculty cohort summary reports were available for faculty. The APRN Exam reliabilities ranged from .91 for the Family Exam to .92 for the Adult-Gerontology Exam.

HESI APRN Exit Exam Practices/Strategies Questionnaire

In Phase I, the HESI APRN Exit Exam Practices/Strategies Questionnaire, a 13-item multiple choice survey, was sent to deans/directors via a private internet SurveyMonkey® link. This survey explored the programs’ testing policies and practice that included: benchmark use, strategies used to prepare for the exam, duration of time allotted to preparation, re-testing practices, remediation used, and consequences of the exit exam.

Certification Outcome Sheet by Student HESI Scores

Phase II included providing deans/directors with a unique Certification Outcome Sheet of Student HESI Scores that was comprised of a list of their school’s students who took either the Family or Adult-Gerontology Exit Exams during academic years 2013 – 2015. The student’s names and exam scores were provided. Deans/directors were asked to add a certification examination pass/fail status for each student who had taken the HESI™ APRN Exit Exam after which they were instructed in how to de-identify the data prior to returning the Phase II survey.

Sample

The sample of participants for the study was drawn from Elsevier’s HESI™ APRN Exit Exam users for Family and Adult-Gerontology APRN Exit Exams located within the US. All nursing school programs were housed within accredited colleges/universities whose graduates were eligible to take a national certification at either the AANPCB or ANCC.  The timeframe included academic years beginning 1 September 2013 through 31 August 2016. One hundred and fourteen (N =114) graduate nursing schools across the US were invited; 15 schools responded to the questionnaire for a 13.15% response rate (See Figure 1). Many deans who opted not to respond indicated that they did not track individual outcomes for the certification exams.

Figure

Distribution of Nursing Schools (N=114) Using Family and/or Adult-Gerontology HESI™ APRN Exam During Academic Years 2009 – 2015.

Informed Consent

Institutional Review Board (IRB) approval was granted through the principal author’s university. Implied consent was applied for all participants who responded to the query and participation survey link. Participants were asked to identify outcomes for students in their program on their first attempt at certification, as well as to answer several questions regarding testing and remediation policies implemented at their schools. To protect subjects’ anonymity, data regarding student certification examination outcomes were reported to the research team as aggregate data only.

Data Collection

Data were collected via the HESI APRN Exit Exam Practices/Strategies Questionnaire and the Certification Outcome Sheet by Student HESI Scores spreadsheet with student’ name, HESI™ APRN Exit Exam scores, and a column to indicate pass or fail on the student’s first attempt on the certification exam. The practice survey was completed in SurveyMonkey®. The spreadsheet was completed with the pass/fail status for each student and returned by e-mail.  The student names were removed by the faculty before returning the spreadsheet and the files were password protected.

Results

Predictive Validity

Results from a national, multi-site, US sample of graduate programs indicated that electronic standardized end of program assessments are highly accurate at predicting APRN certification success. End of course APRN specialty exams were utilized by faculty to determine their students’ standings compared to the national sample. First attempt certification outcomes were reported for 497 APRN students. Students’ scores varied from 315 to 1168 on the Adult-Gerontology NP exam with a mean score of 736. Scores on the Family NP exam varied from 366 to 1246 with a mean of 752. Twenty-seven percent (n=132) had scoring categories of A/B (900+) and D (800-849) and these students were 100% successful on certification examinations. Scoring category C (850-899) had the smallest number of students (n= 40, 8%) with two students who were unsuccessful first-time test takers which accounts for a 95% predictive rate. The majority of students (n= 325, 65%) scored in categories E/F (700-799) and G/H (<699) which predicted certification passage 94% of the time. Students with Scores of 900+ predicted pass rates on the Adult-Gerontology NP and the Family Certification exams 100% of the time, as did those students in 800-849 scoring category.  These results, support continued predictive validity of the HESI APRN Exit Exam. See Table 1 and 2 for specific rates per scoring category.

Table 1

Number and Percent of Students by Scoring Category and HESI APRN Family and Adult-Gerontology Exam Scores

Table 2

Predictive Accuracy of HESI APRN Family and Adult-Gerontology Exam Scores by Scoring Category (N = 497)

Curriculum Assessment

End of course APRN exams were utilized by faculty to determine their students’ standings compared to the national sample. The results also assist faculty to identify cohort concept or content weakness for appropriate remediation, as well as provide input into curricular revisions. In response to the survey, 40% of the deans/directors (6) indicated that the HESI APRN Exit Exam counts in the student’s course grade. Seventy-five percent (6) of the 8 respondents who answered the question on percent of the grade reported that the exam counts in the range of 10% to 50% of the course grade. Forty-percent (6/15) of the schools reported having an APRN exit exam policy in place. Eight schools specified a benchmark score with one (12.5%) each indicating 700, 750, or 850 and three (37.5%) using a score of 800 as the benchmark score.

Consequences

Eleven respondents also described consequences if students failed the exit exam. Respondents could select multiple options. The most common consequences included: 18% (3) course failure, 18% (3) delay or deny graduation, 18% (3) delay or deny a program completion letter. Sixty-four percent (7) utilized other consequences for failure to meet the benchmark including remediation, completion of a second exam, and completion of a written comprehensive exam.

Remediation Strategies

Eleven of the 14 schools (79%) reported that remediation strategies to facilitate certification success were in place. The most common strategy for remediation used was a live certification preparation course at 67% (8). Other approaches included faculty review sessions (58%; 7), the use of an advanced practice review textbook (50%; 6), and other computer-based tutoring (33%; 4). Sixty-nine (9) percent of the schools recommended remediation after exit exam failure, while 31% (4) required remediation. Sixty-seven (8) percent of schools provided a test review after the exam was given. Immediate review of the test by individual students using the online answers and rationales was the most common form of review methodology. Finally, the HESI APRN Exit Exam is primarily administered near the end of the program by 80% of the schools, 92% (11) of the schools use the HESI APRN Exit exam to evaluate curriculum, and 67% (8) use the results to change course content.

Conclusion

A national multi-site US sample of graduate family and adult-gerontology APRN programs found the HESI APRN Exit Exams highly accurate at predicting certification success. For this study, a benchmark score of 800+ was highly predictive of certification success. Within A/B – D scoring categories, only two students were unsuccessful on first attempt (n=172, 98.8%) examination. The findings of this study are consistent with the test producer’s preliminary benchmark setting studies for the HESI APRN Exit Exams that set a minimum score of 750 or higher. Faculty can set school benchmarks to meet their program’s mission, vision, and objectives. Using a benchmark of 800+ corresponded with 98.8% certification success rate in this sample. The 94% predicted certification pass rate (n=325) for the lower categories E/F and G/H suggests that HESI Exit Exam preparation, the test taking experience, and scores may improve certification success over the national certification rates ranging from 80% to 88%.

Graduate schools can use standardized testing products to assess for content mastery throughout and at the end of APRN programs. This study focused on end of program mastery testing that predicts the attainment of national certification. From the theoretical perspective of the Reconceptualized Vygotsky’s Theory of Goal Attainment, implementing graduate standardized testing can provide structure and process for expert faculty/preceptors (first voice) and students (second voice) to identify the knowledge gaps from the exit exam outcomes (third voice).9 Faculty are challenged to develop and foster individualized student-centered learning within the last few weeks of the student’s course of study. This challenge is intensified when coupled with large class sizes, online course delivery, and accelerated program formats. This study found that the most frequently used end of course remediation was: faculty review sessions or commercially produced face-to-face, on-line, or audio-video-based content specialty courses. One approach commonly used by undergraduate faculty is consecutive testing, which provides multiple testing experiences, feedback, and remediation time for content mastery. Repeat graduate standardized testing should be tracked and trended by programs to determine if consistent practices improve content mastery. Establishing a correlational rate for each teaching/learning strategy initiated by faculty and certification success is also needed.

Further research regarding the utilization of APRN standardized testing should determine the relationship of individual student first time pass rates with specific remediation strategies implemented by nursing programs. Such research outcomes might be accomplished by school collaborations or large national educational system school investigations. Testing summary reports generated by the test producer provide faculty the evidence for faculty course or curricular evaluation. Results also demonstrated that few schools had standardized testing policies in place, established benchmark scores, or counted standardized exams as part of course grades.

Limitations

Standardized testing is one of multiple assessment strategies faculty use to validate APRN students’ knowledge, skills, and competencies. During the student’s program of study, testing, as well as simulations, case studies, and clinical experiences provide student learning opportunities over time. Consequently, the generalizability of standardized testing is limited by the multiple strategies programs simultaneously implement and timing of standardized testing within the course of study. The national APRN certification bodies (AANPCB and ANCC) report aggregate student outcomes to nursing programs annually, which limits the faculty’s ability to determine which students pass/fail the certification examination on the first trial. Therefore, correlational data cannot be calculated for each of the strategies offered by the various programs. While this study has the largest number of participants of the previously reported HESI APRN Exit Exam predictive studies (N = 497), the study outcomes are limited by the small number of participating schools (N=15). When viewing this study from a theoretical framework perspective, a gap exists. In the Reconceptualization of Vygotsky’s Theory of Goal Attainment by Eun, et al,9 first voice is represented by faculty and the introduced third voice conceptualized as standardized testing results was well represented, however, second voice, that of the student was not represented in this study. Further exploration regarding the learning and study strategies students utilize when taking standardized exams will assist in the overall advanced practice educational delivery in preparation for certification success.