Continuing the Conversation on Program Rankings

By James F. Cawley, MPH, PA-C
APAP President

Rankings of graduate-degree-granting PA educational programs by U.S. News & World Report is squarely on the agenda of the APAP Board of Directors. Members attending the Hot Topics session at the fall 2002 Educational Forum expressed opinions and concerns on this issue and, since that time, the board has considered the impact the rankings have on perceptions of PA program stature and quality. The focus of the board’s attention, in particular, has been on the impressions prospective applicants derive from the rankings and, more generally, on whether to develop a formal organizational policy statement on program rankings.

Critics of the rankings have expressed concern for what they view as methodologic shortcomings of the survey, as conducted by U.S. News & World Report. Surveys are sent to the program directors and senior faculty members, who are asked to rank on a 1 through 5 scale — with 5 being the highest — the programs in their discipline. The respondents rate the academic quality of PA programs based on scholarship, curriculum, and the caliber of the faculty and graduates. Scores for each program are averaged across all respondents who rate the programs. The “reputation scores” for this third cycle of the survey of PA programs were collected in November 2002. Every program’s performance is presented relative to the other programs with which it is being compared. Only graduate PA programs with scores of 2.5 or above (n = 63) were published in the third survey; 70 percent of eligible APAP member programs responded to the survey.

For some PA educators, particularly those critical of the rankings, the survey is viewed as a popularity contest, which may or may not reflect the true quality of programs. Academic institutions have been ranked by reputation and other attributes since 1924. U.S. News & World Report began ranking academic institutions in the early 1990s, and its rankings report has become one of the most popular lists currently in use by individuals with an interest in comparing educational programs.

Rankings of American medical schools published annually by the newsmagazine U.S. News & World Report are widely used to judge the quality of the schools and their programs. In a sharp critique of the newsmagazine’s rankings of medical schools, McGaghie and Thompson blast the rankings on methodologic and conceptual grounds, arguing that the annual U.S. News medical school evaluations fall short in both areas. Three categories of program quality indicators — different from those used by U.S. News — are presented in the paper as alternative ways to judge medical schools. The authors conclude that the annual U.S. News & World Report rankings of American medical schools are ill-conceived, unscientific, and poorly conducted; they ignore medical school accreditation; judge medical school quality from a narrow, elitist perspective; and do not consider social and professional outcomes in program quality calculations. McGaghie and Thompson note that medical school rankings have no practical value as published in the newsmagazine and fail to meet standards of journalistic ethics.¹ There are those in PA education who feel that these findings would apply as well to the U.S. News rankings of PA programs.

Others make the point that there are many programs spanning a wide variety of disciplines that appear in the rankings of U.S. News & World Report, and these include graduate programs in business administration, education, and law, as well as numerous disciplines within the health professions, such as physical therapy, pharmacy, and nursing. It is also well known that a number of institutions sponsoring highly ranked PA programs promote the survey results extensively in their promotional and student recruitment materials.

There is no organizational consensus regarding the rankings. A survey of PA educators’ opinions on the rankings was conducted by Blessing and colleagues shortly after the second survey appeared in U.S. News & World Report. The Blessing survey asked PA program directors to identify program characteristics that would best objectify reputations. One hundred and twenty-six program directors were mailed a 75-item survey in the fall of 2000 and 95 responded (75%). The most notable aspect of the survey results was the lack of agreement on which PA program attributes should be measured. The greatest agreement was on factors such as faculty-to-student ratio, graduation rate, student attrition rate, and PA National Certifying Examination (PANCE) scores. Agreement was nearly unanimous among responding programs that program rank should not be a component of the accreditation process. Despite the human need to compare and contrast, there is little agreement among PA program directors on the elements that should determine a program’s ranking. Without near-unanimous support across PA programs for internally developing objectives by which to rank programs, Blessing et al. concluded that this aspect of PA education is best left to outside agencies and organizations. Thus, there appears to be a distinct ambivalence among PA educators with regard to the rankings.²

The APAP board continues to discuss the appropriate organizational strategy to deal with the U.S. News & World Report rankings. One approach being considered is the development of a frequently asked questions (FAQ) document designed to educate and inform prospective PA applicants of the multiple, and sometimes complex, indicators of PA educational program strengths, while emphasizing that all PA programs have discrete strengths and advantages. It should be noted that APAP has taken no official position either for or against the U.S. News & World Report rankings at this time and continues to monitor organizational opinion.

References:

1. McGaghie, WC, Thompson, JA. America’s Best Medical Schools: A Critique of the U.S. News & World Report Rankings. Acad Med 2001 76: 985-992.

2. Blessing, JD, Hooker, RS, Jones, PE., Rahr, RR. An Investigation of Potential Criteria for Ranking Physician Assistant Programs. Perspective on Physician Assistant Education 2001 12(3): 160-6.

Main