Medical Education

Jun 2015

Assessing Competency in Physical Medicine and Rehabilitation Residency: the ACGME Milestones Initiative

Julian Willoughby, MD, MPH, Vu Nguyen, MD, MBA, and William L. Bockenek, MD
AMA J Ethics. 2015;17(6):515-520. doi: 10.1001/journalofethics.2015.17.6.medu1-1506.

 

The profession of medicine has a unique ethical obligation to ensure that it trains new practitioners to be competent and ready to serve patients who are often in a vulnerable state and unable to assess the competence of the practitioners who are treating them. For many years, the proof of a physician’s competency has been relatively simple and process-oriented—if a physician passed a board exam and/or completed the residency training, then he or she was de facto deemed competent.

The traditional approach to assessment began to change in 1999, when the Accreditation Council for Graduate Medical Education (ACGME) created the six clinical competencies—patient care, medical knowledge, practice-based learning and improvement, interpersonal communication skills, professionalism, and systems-based practice [1]—that provide a framework for outcomes-focused guidelines specifying what it means to be “clinically competent.” About a dozen years later, the ACGME began a second step in the process—the Next Accreditation System (NAS)—to create detailed, specialty-specific knowledge and skill milestones by which to assess a trainee’s progress toward competency.

Why the NAS?

Historically, competency as a well-rounded physician was inferred when residency was completed. Board exams provide a more objective measure of knowledge—but to what extent can written exams evaluate what it means to be a doctor? One may argue that board tests primarily evaluate just one of the ACGME core competencies, medical knowledge. The other five competencies are much more difficult to assess with standardized testing. For these, many specialties have relied on supervising clinicians’ written evaluations and Likert scales, methods that are highly subjective. Others, including physical medicine and rehabilitation (PM&R), have used additional standardized oral exams to bolster assessments. According to the American Board of Physical Medicine and Rehabilitation (ABMPR), “the Part II [oral] exam indeed measures something different than the Part I [written] exam…the oral examination is structured to measure the ability of the candidate to apply medical and physiatric knowledge and skill in patient care, interpersonal and communication skills, professionalism, and systems-based practice” [2]. However, there are limitations to a one-time oral exam performed at the end of resident training.

None of these methods provides a consistent or comprehensive picture of trainee competency. While the ACGME’s core competencies identify the areas in which physicians should be trained, they cannot and are not intended to identify specific activities needed for assessing competency during residency training in a specific specialty. Certain specific skills are unique to physiatrists, for example, such as understanding the complications of and how to treat patients with spinal or brain injuries, which may not be a priority of a resident training in another specialty. Thus, as part of the Next Accreditation System, residency programs were asked to create competency-based milestones that are specialty-specific and provide standardized, objective, and reproducible data on the physician-in-training’s progress toward becoming an independent practitioner [3].

Milestones, as defined by the ACGME, are “competency-based developmental outcomes (e.g., knowledge, skills, attitudes, and performance) that can be demonstrated progressively by residents and fellows from the beginning of their education through graduation to the unsupervised practice of their specialties” [4]. In short, the milestones articulate, in very specific terms, the physician-in-training’s demonstrable progress toward becoming an independent practitioner who is capable and, most importantly, competent in the specific tasks deemed crucial by experts in the field. The milestones for physical medicine and rehabilitation residencies were developed over two years by a working group composed of nine current or former members of the ACMGE PM&R residency review committee (RRC), including a resident representative, directors of the American Board of PM&R, and residency program directors.

How does the ACGME envision that the milestones program will improve residency training? The milestones should aid in accreditation by:

  • allowing for continuous monitoring of programs, thus increasing time between time-consuming site visits conducted by volunteer committee members,
  • providing national aggregate competency outcomes for each specialty,
  • creating a community focused on evaluation and research [4].

For education, the milestones will:

  • provide a detailed framework for evaluation of competency,
  • guide curriculum development of residency programs,
  • support better assessment programs,
  • aid in early identification of struggling residents and fellows [4].

And for individual trainees, the milestones will:

  • “provide more explicit and transparent expectations of performance,”
  • “support better self-directed assessment and learning,”
  • “facilitate better feedback for professional development” [4].

Additionally, to ensure that milestone evaluations are reported and reviewed, the NAS mandates that each residency program form a clinical competency committee (CCC) “comprising three or more members of the active teaching faculty,” which may include the program director as well as other relevant staff [5]. The ACGME has provided a comprehensive guide on how the CCC should be organized and implemented across all programs, thus ensuring some uniformity in applying the milestones in trainee evaluation. But appropriate flexibility in the constitution of the CCCs among programs and specialties is also allowed. For example, because PM&R programs emphasize teamwork with therapists and other members of the health care team, they may include some of these specialists in their CCC. In this way, residents’ progress toward the milestones is evaluated in light of both their level of training and the environment in which they are training. This system of specialty-specific milestones and program-specific CCCs provides a flexible framework within which programs can guide curriculum development and apply the milestones in the assessment of their trainees.

The milestone initiative is an ambitious undertaking that seeks to significantly improve the process of assessing medical resident and fellow competency. Applied through the CCCs, the milestones, should fill the assessment gap by providing systematic, comprehensive, and specialty-specific evaluation of ACGME competencies throughout the course of resident training.

Potential Pitfalls

There is as yet little objective data on whether the milestones actually provide the intended benefits. In 2012, the members of the PM&R working group piloted the use of milestones in their own programs. Feedback regarding the clarity of the evaluated milestones was positive and constructive criticisms were offered. Respondents from larger programs were chiefly concerned that their programs would need more time to complete the evaluation process than would smaller programs, where members of the CCC were likely to be more familiar with residents’ performance. It was also noted that commonly used assessment tools such as post-rotation surveys were not adequate to fully assess the milestones and that additional faculty development would be needed to train faculty to accurately observe and rate resident behaviors [6].

The milestone system may not solve all the extant problems in resident evaluation. It does advance the process of resident evaluation beyond subjective measurement scales and brief comments by supervising physicians—it provides public accountability on a national scale, frequent feedback to guide curriculum development, and more explicit expectations for residents to use in directing their own professional development. However, a significant ethical dilemma is still inherent in both the old and new processes—that is, residency programs’ being in charge of evaluating their own residents. Because residency programs want to appear to train excellent physicians, there may be a conflict of interest in a program’s reporting the quality of the physicians it is producing. The quality of a program’s graduates may affect its funding, recruitment, and stature. Furthermore, the world of residency training is unlike most other workforces because, if a resident withdraws or is dismissed from a program, the workload of other residents and faculty physicians is significantly affected, and it is usually difficult to replace the lost support until the following match cycle.

This conflict of interest can be seen as both a positive and a negative force. It may be that a program with a poorly performing resident would be motivated to remediate and improve that resident’s performance, or it may be that the program would simply artificially inflate the resident’s evaluations to keep from losing an important team member within the workforce until he or she left in a year or so. There is hope that, with the new milestones, a resident’s failure to progress will be identified earlier in the process of training, allowing earlier remediation and a greater chance of resident success. The milestones could thus prevent programs from having to release a trainee but would still rely on them to give honest assessments.

The milestone system should also address the problem of variation in evaluations between programs by having experts in each specialty work together to create specialty-specific milestones that represent an ideal training development path for their residents. The field of PM&R did this by mobilizing voices from all corners of the field, thus taking the onus off individual programs to create shared goals and incentives to advance the specialty as a group. However, even if the milestones are shared among programs within the specialty, there can still be significant variation in how programs apply them to their cohort of residents.

For example, variation is bound to exist in the makeup and role of the CCCs. In the PM&R pilot study, it was noted that, in the larger programs, the CCC members were not completely familiar with all of the residents being evaluated. In lieu of familiarizing themselves with each resident’s performance and filling gaps in their knowledge about that learner, evaluators may merely assume that a trainee’s performance in one area, with which they are familiar, indicates performance in another. This could create a favorable or unfavorable disposition to the resident’s evaluation. A resident’s superior performance observed in one clinical area may well favorably skew the evaluation of that resident in an unobserved area—the “good guy syndrome” [1]. On the other hand, a resident’s less-than-stellar performance in one observed area may be considered representative of his or her performance in all unobserved areas. A given program may not have the resources or staff to assess its resident cohort as well as another program’s CCC and thus may create a situation in which the entire program’s assessments are based more on assumptions than on sufficient information. Although this possibility certainly exists in current assessment methodologies, one must consider whether the increased resources and time needed for each program to implement the milestones and CCC could further increase the variability in assessment of competency.

Conclusion

Medical educators share a moral obligation to provide competent care for the public and, therefore, an obligation to train competent clinicians for the future. The milestone initiative seeks to do just that by reforming the process of assessing competence. It alters not only how we assess competence, but also how we define and achieve it. Milestones will move graduate medical education from a process of completing rotations and years of training to one of demonstrating and perfecting the skills associated with one’s specialty. Making assessment more measurement based and objective will focus trainees’ and curriculum designers’ efforts on areas of weakness. Programs will be able to develop and reform based on objective data, and residents will have a clearer picture of the expectations they will need to meet to graduate. Pitfalls do exist but are less profound than those of prior evaluation schemes. Challenges may include the allocation of additional resources needed to implement CCCs. So far, we believe that the potential benefits far outweigh the challenges.

The milestone initiative will most certainly improve the assessment of competency in residency training in PM&R and other specialties. The full realization of milestones is still far away, but, by enhancing the evaluation of current residents and fellows, graduate medical education will continue to improve, and it will ultimately improve the lives of those we are sworn to heal.

References

  1. Ling L, Derstine P, Cohen N. Implementing milestones and clinical competency committees. Accreditation Council for Graduate Medical Education. April 24, 2013. https://www.acgme.org/acgmeweb/Portals/0/PDFs/ACGMEMilestones-CCC-AssesmentWebinar.pdf. Accessed April 15, 2015.

  2. American Board of Physical Medicine and Rehabilitation. New part II examination format. Diplomate News. 2005:6. http://www.abpmr.org/newsletters/documents/summer2005diplomatenews_002.pdf. Accessed April 15, 2015.

  3. Carter WA Jr. Milestone myths and misperceptions. J Grad Med Educ. 2014;6(1):18-20.
  4. Accreditation Council for Graduate Medical Education. Milestones. https://www.acgme.org/acgmeweb/tabid/430/ProgramandInstitutionalAccreditation/NextAccreditationSystem/Milestones.aspx. Accessed April 15, 2015.

  5. Andolsek K, Padmore J, Hauer KE, Holmboe E. Clinical competency committees: a guidebook for programs. Accreditation Council for Graduate Medical Education; 2015:2. https://www.acgme.org/acgmeweb/Portals/0/ACGMEClinicalCompetencyCommitteeGuidebook.pdf. Accessed April 15, 2015.

  6. Bockenek WL, Massagli TL, Swing SR, Fischer C; the PMR Milestone Working Group. The development of the physical medicine and rehabilitation milestones. J Grad Med Educ. 2014;6(1)(suppl 1):204-206.

Citation

AMA J Ethics. 2015;17(6):515-520.

DOI

10.1001/journalofethics.2015.17.6.medu1-1506.

The viewpoints expressed in this article are those of the author(s) and do not necessarily reflect the views and policies of the AMA.