Using practice analysis to improve the certifying examinations for PAs

A practice analysis is a tool that bridges knowledge and clinical performance into a format that permits assessment. For physician assistants (PAs), this contributes to a psychometrically sound examination administered by the National Commission on Certification of Physician Assistants (NCCPA). The 2004 practice analysis of 5,282 completed PA surveys (13.4% out of 39,517 sent) was representative of the PA population in years experience, geographical distribution, and practice specialty. The survey revealed 8 content domains with formulating the most likely diagnosis, basic science concepts, and pharmaceutical therapeutics as the three skills needed for most scenarios. The data were also analyzed by patient acuity (acute limited, chronic progressive, life-threatening emergency). As a result, NCCPA's test item pool and content blueprint for assessing core knowledge of American PAs on the Physician Assistant National Certifying Examination (PANCE) and the Physician Assistant National Recertifying Examination (PANRE) has been enhanced.

What do physician assistants (PAs) do, how skilled are they in performing these activities, and how important are these activities? For many professions, a practice analysis provides the link between examination content and real-world performance. A practice analysis is a systematic plan to study a profession and a critical step in developing a psychometrically sound and realistic credentialing examination. The goal of practice analysis is to construct a detailed description of the knowledge and skills needed in an occupation. Information obtained from members of the profession provides the framework for examination development, including the creation of test specifications, item writing, and examination construction.

The documentation of job-related knowledge, skills, and abilities is referred to as content validity. Establishing content validity requires identifying the knowledge and skills PAs use in their daily practice, the frequency of the activities that require that knowledge and skill, and how critical those activities are compared to minimally-competent performance. The National Commission on Certification of Physician Assistants (NCCPA) periodically undertakes this critical endeavor linking knowledge to clinical practice to facilitate the continuous development and refinement of the Physician Assistant National Certifying Examination (PANCE) and the Physician Assistant National Recertifying Examination (PANRE). Since the knowledge and skills of a profession change over time, a new practice analysis must be undertaken periodically. The need for a new practice analysis is dependent on the dynamics of the profession being assessed.

The practice analysis is, thus, a critical part of NCCPA's effort to ensure that its examinations are consistent with the definitions of PA competence identified in the Competencies for the Physician Assistant Profession, which is a public document available at www.nccpa.net/PAC/Competencies_home.aspx. Of the six PA competencies, NCCPA's examinations serve to primarily assess medical knowledge.

Although some might believe a practice analysis should survey most of the certified PA population for which information is available, this is neither feasible nor optimal. The over sampling of experienced practitioners can skew a picture of clinical practice that might differ from those with relatively little clinical experience. Instead, the goal is to create a model-based practice analysis and test specifications advocated by a number of researchers.1-3

The NCCPA has undertaken role delineation studies since 1976 as a means of validating the examination content for the PANCE and the PANRE.4 The intent of the 2004 study was to serve as an empirical reference for NCCPA in its review of examination content. The study's objectives were to

1. Create an array of important knowledge and skill statements by an expert panel
2. Confirm the adequacy of that array by practicing PAs
3. Account for the contextual impact of PAs' judgment of importance
4. Interpret the findings in terms of examination recommendations.

These studies also serve as historical landmarks in identifying PA activity in the United States; and as such, the study is outlined here.

METHOD

Survey instrument The NCCPA and the National Board of Medical Examiners jointly conducted a Web-based practice analysis survey in 2004. The survey consisted of five primary sections: an invitation e-mail, an informed consent form, a background information section, a knowledge and skill statements section, and an open-ended question for additional comments. Background questions solicited information regarding current employment and specialty. Participants who did not agree with the informed consent form and/or who were not working as PAs were excluded from the survey.

A 15-member panel of content experts consisting of PAs and educators discussed clinical traits essential for safe and effective practice, the knowledge and skills required to master these clinical traits, and the effect of various contexts on the importance of these tasks. A total of 118 knowledge and skill statements emerged; 85 of which included types of knowledge and 33 of which included specific skills important for safe and effective practice.

A pilot study ranked the 118 knowledge and skill statements in terms of importance to PA practice and by patient setting. The statements were ranked for importance using a 4-point scale, including 1 or almost no importance, 2 or little importance, 3 or moderate importance, and 4 or very high importance, and by patient setting, including ambulatory (outpatient office or clinic), inpatient (hospital), emergency (emergency department), and surgery (operating room).

Feedback from the first pilot survey served to refine the survey instrument and process. First, because of a low (7.7%) response rate, researchers determined through participant comments that the survey's length may have contributed to the poor response rate. In addition, participant feedback suggested modifications to the rating scales.

For a second pilot study, four surveys—with equally distributed content—were created with 30 knowledge and skill statements on each form. In addition, a 5-point importance skill for the knowledge and skill statements was established where 1 equaled no importance, 2 equaled little importance, 3 equaled moderate importance, 4 equaled high importance, and 5 equaled critical importance. Further, the four patient settings were replaced with three patient acuity levels: acute limited, chronic progressive, and life-threatening emergency.

The second pilot study had a response rate of 11.4%. Verbal and written feedback on the content of the knowledge and skill statements was positive. In addition, participants showed more discrimination on the importance and patient acuity scales. As a result, the survey instrument was adopted.

Survey The final Web-based survey was distributed in August 2004 to 39,500 valid e-mail addresses of PAs. Participants in the pilot studies were excluded.

Data analysis Mean importance ratings were calculated and compared at the content domain level by patient acuity. An overall multivariate analysis of variance (MANOVA) within-subjects design included eight dependent variables (F [8,16] = 50,810.80, P < .001). This indicated that the mean importance ratings across the 8 content areas varied as a function of patient acuity. To test the statistical significance of the differences in mean importance ratings, eight separate one-way repeated analyses of variances (ANOVA) were conducted for each content domain with importance rating as the dependent variable and patient acuity as the withinsubjects factor. All effects were statistically significant at the .001 significance level.

RESULTS

Of the 39,517 surveys sent to valid e-mail addresses, 5,282 (13.4%) were completed. Among the respondents, 36% had practiced as a PA for 12 or more years, 23% from 6 to 11 years, and 29% from 2 to 5 years. Only 12% of the respondents had practiced for less than 2 years (see Figure 1). The vast majority (85%) of survey participants worked full-time in clinical practice (see Figure 2). Geographical location, practice setting, and employing physician's specialty varied. Twenty-four percent of respondents indicated their practice setting as single specialty group practice, while 19% reported hospital practice (see Figure 3). Two-thirds of respondents were located in urban or suburban communities (see Figure 4).

The range of specialties reported in the study population reflects the PA population at large. The majority of survey participants indicated that their primary specialty was family or general medicine (29%), followed by “other” (19%) and emergency medicine (12%) (see Figure 5).

Importance ratings The mean importance ratings by content domain ranged from 3.5 (health maintenance) to 4.3 (formulating the most likely diagnosis). Health maintenance statements fell between moderate importance and high importance, whereas formulating the most likely diagnosis statements fell between high importance and critical importance. All mean importance ratings were greater than 3.0, on a 5.0 scale, suggesting that all content domains were considered of moderate importance or higher. The mean importance ratings are across all three levels of patient acuity (see Table 1).

While all content domains are important for safe and effective PA practice, the study data suggest the eight content domains can be listed in terms of the degree to which they are important relative to each other. In other words, while participants judged all content areas to be important, some content areas were judged to be more important than others.

In addition, the data were also analyzed by patient acuity (acute limited, chronic progressive, life-threatening emer gency). Table 2 presents the content domain ranking for each level of patient acuity. For patients with acute limited conditions, participants rated formulating the most likely diagnosis items as the highest importance to safe and effective practice and health maintenance as the least important. For chronic progressive patients, participants considered pharmaceutical therapeutics items the most important and professional practice items the least important. In life-threatening emergencies, formulating the most likely diagnosis statements were rated the most important and health maintenance statements were rated the least important.

Analysis of variance The results of the overall MANOVA indicated that the main effect of the between-subjects variable, content domain, is significant (F [8,16] = 50,810.80, P < .001). This finding suggests that the variation in mean importance ratings across the eight content domains represents statistically significant differences in level of importance. Additionally, the MANOVA results show that the main effects of the within-subjects variable, patient acuity, is significant (F [8,16] = 538.25, P < .001). This suggests that the mean importance ratings across the eight content areas also vary as a function of patient acuity.

Open-ended comments At the end of the survey, participants were provided the opportunity to make comments regarding either the knowledge and skill statements or the survey itself. Of those completing the survey, 23% (1,239 participants) responded. The comments were conceptually categorized, and it was determined that participants' comments were relatively consistent with well-established discussion points within the profession.

DISCUSSION

This role delineation study identified knowledge and skills that define the function of a clinically active PA. It captures the breadth and scope of PA activity in a wide variety of settings. Such occupational analyses are undertaken by those both within and beyond healthcare and are generally considered essential for licensure.3 The NCCPA has undertaken this charge since 1976; fulfilling a responsibility to periodically assess PAs in contemporary medical practices. Since the PA profession continues to evolve, the periodic undertaking of a practice analysis helps to keep the NCCPA's examination content blueprint an accurate reflection of PA practice. The 1998 practice analysis provided the fundamental content blueprint for current use in NCCPA's PANCE and PANRE, and each subsequent analysis improves the process of understanding what PAs do and what skills are critical for competent practice.5 Each analysis seeks to create lists of knowledge and skills that are representative of the actual clinical practice of PAs. Although this study had a modest 13.4% response rate, the distribution across practice setting and specialty was representative of the PA census and considered valid. As a result, the study not only authenticates the content of the examinations but also validates to the American public and employers that PAs are practicing in a manner consistent with contemporary medical care. In turn, practice analysis guides educators as to what are the fundamentals of medicine that all PAs should know.6,7

Since the majority (85%) of participants work full-time in clinical practice and have worked anywhere from 2 years to more than 3 decades, the study's findings are based on responses from PAs with the clinical experience required to rank the importance of various types of knowledge and skills necessary for safe and effective practice. The PA respondents also represent a full spectrum of practice settings and community types.

The narrow standard deviation in mean importance ratings across content domain and patient acuity represents a remarkably consistent set of shared values that PAs place on these subjects. It appears that PAs believe that patient acuity is an important factor in determining the criticality of certain types of knowledge and skills essential for safe and effective practice. When the results of the eight separate one-way ANOVAs are examined, they too reveal statistical significance at the 0.001 levels. Table 3 illustrates the mean importance rating by content domain and patient acuity. These findings regarding the relationship between content domain, patient acuity, and level of importance to safe and effective practice are considered highly significant for almost all clinically active PAs.

LIMITATIONS

Limitations to this study are no different than any other selfreport survey: Do the findings reflect the real belief and action of the individual? The survey response rate (13.4%), while modest, is not atypical for practice analysis studies. However, the survey participants appear to be representative of the national population of PA practitioners when compared to the AAPA Census undertaken in 2004.8

Whenever expert panels are charged with developing lists of knowledge and skill statements, it is expected that most statements will be important. Excluding unimportant or unessential statements is part of the task. As a consequence, when the highly refined list is presented to a survey population, few statements tend to be valued as unimportant. Thus, the respondent is asked to rank order the statements, and the researcher then identifies the significant findings deemed important by study participants. The present study was not immune to this limitation, and many differences were decid ed at the two decimal level. Nevertheless, the data do display variability in judged importance, and this may be attributed to the decision to incorporate patient acuity as a second dimension. As a result, NCCPA's item pools are now coded according to the patient acuity dimension.

As a footnote to this process, the distribution of the patient acuity construct across a 300-item PANCE form was coded by content experts according to the three levels of patient acuity used in this practice analysis. Each item received a patient acuity code of acute-limited, chronic progressive, lifethreatening emergency, or not applicable (N/A) (where the experts were unable to classify the item).

At the conclusion of the practice analysis, the report was provided to the NCCPA Board of Directors for consideration. While the current content percentages were maintained, the analysis helped to more narrowly define and appropriately categorize the content included in each content area.

CONCLUSION

A practice analysis is a critical function of a dynamic health profession that has the care of society entrusted to them. The earliest directors of the NCCPA saw fit to mandate an examination for entry-level practice and a recurring examination to ensure that PAs are continuing to practice in a safe and effective manner. Since PAs provide health care through a broad spectrum of activity, the NCCPA is challenged to develop examinations that reflect PA practice appropriately and accurately. The 2004 practice analysis improves the method for identifying and ranking the importance of knowledge and skills in various areas. Based on this study, patient acuity is now incorporated into the NCCPA examination process.

NCCPA continues to be proactive in reflecting PA practice trends on its examination, and the next practice analysis will be conducted in 2009 with results reported in 2010. One topic that may be quantified by the study is the role of genetics in primary care practice. While NCCPA's exams have long reflected the genetic component of various diseases and disorders, the emergence of genetics in the practice analysis may drive modification of the content blueprint to reflect the changing reality of PA practice.

NCCPA's periodic analysis of PA practice reflects its commitment to further refine its processes and to ensure that the knowledge and skills for safe and effective practice are accurately reflected in its certifying and recertifying examinations. JAAPA

REFERENCES

1. Kane M. Model-based practice analysis and test specifications. Applied Measurement in Education. 1997;10(1):5-18.

2. Scoles PV, Hawkins RE, LaDuca A. Assessment of clinical skills in medical practice. J Contin Educ Health Prof. 2003;23(3):182-190.

3 Raymond MR. A practical guide to practice analysis for credentialing examinations. Educational Measurement: Issues and Practice. 2005;21(3):25-37.

4. Hooker RS, Carter R, Cawley JF. The National Commission on Certification of Physician Assistants: history and role. Perspective on Physician Assistant Education. 2004;15(1):8-15.

5. Cawley JF, Andrews MD, Barnhill GC, et al. What makes the day: an analysis of the content of physician assistants' practice. JAAPA. 2001;14(5):41-56.

6. Jones PE. Physician assistant education in the United States. Acad Med. 2007;82(9):882-887.

7. Cawley JF. Physician assistant education and PANCE performance: a passing controversy? Perspective on Physician Assistant Education. 2002;13(2);79-80.

8. American Academy of Physician Assistants. 2004 AAPA physician assistant census report. http://www.aapa.org/research/04census-intro.html. Accessed January 13, 2008.