AMA Journal of Ethics®

Illuminating the art of medicine

Journal of Ethics Header

AMA Journal of Ethics®

Illuminating the art of medicine

Virtual Mentor. December 2012, Volume 14, Number 12: 937-945.

ETHICS CASE

  • Print
  • |
  • View PDF

Personality Testing in Resident Selection

Is it fair to have applicants to residency programs take personality tests? Are personality tests effective predictors of performance as a physician?

Asher Tulsky, MD, Julie M. Aultman, PhD, and Matthew J. Zirwas, MD

Brent, a fourth-year medical student applying to residencies in internal medicine, has worked diligently during medical school, achieving high grades, garnering commendations for empathic bedside care, assuming leadership positions in several organizations, and volunteering in underserved care programs. He is a competitive applicant for the specialty and has already received a number of interview requests. His first interview is at a small, exclusive, but well-known program, and Brent is feeling nervous but excited after having achieved what he had thought was his best.

After the program director introduces himself and the program, a personality test is administered to all of the applicants. Brent begins to feel apprehensive; why is this test being administered? He is worried that his chance of matching at such a program could be ruined if he doesn’t give the “right” answers to these questions that, as far as he can tell, aren’t even pertinent to medicine. Brent’s discomfort about the personality test rattles him to the point that it affects his mood and candor during the interview.

Upon returning home, Brent and his classmates discuss what they have encountered on the interview trail, from colorful interviewers to student faux pas. He tells them about the personality test: “I don’t see the importance of this personality test. Shouldn’t my application speak for itself? It makes me so mad that I might not match there because of how I did on some ridiculous personality test. Seriously, were they interviewing me to see if I’d make a good doctor, or to see if they wanted to date me?”

Commentary 1

by Asher Tulsky, MD

In this scenario, Brent, a fourth-year medical student, is confronted with an unanticipated request to complete a personality inventory as a part of his interview day for a residency position. Fearing that a ten-item personality “test” may undermine his acceptance into a program that, based on his application, he is well qualified for, he laments that the interview questions should be sufficient. Unfortunately, as any program director will attest, the typical interview process is not all that effective in consistently finding the best fit.

Generally, applicants’ medical knowledge and patient care skills are well addressed in their clerkship performance and USLME scores, both generally well summarized in the medical student performance evaluation (MSPE) provided by the dean of students’ office. Much more difficult to determine are a student’s noncognitive qualities, such as conflict management, communication with ancillary staff, and professionalism. While clerkship grades are assumed to include an assessment of these attributes and some medical schools now provide a separate professionalism summary, these are difficult skills to assess well without specific training and focused attention. The halo effect, or the influence of one aspect of performance on the grading of others, undermines the validity of such ratings.

Does is it make sense to even try to assess these characteristics of applicants? Grades and standardized test scores should in theory be sufficient to identify qualified and competent applicants, and certainly meeting with faculty and residents during the interview process should reveal most serious potential problems. The value of data from these sources in predicting performance, however, is modest [1, 2]. Further, the medical education literature suggests that a significant minority of residents have serious problems during their training. The estimate was 7 percent of residents in one national survey of all program directors [3] and another, by the American Board of Internal Medicine, estimated that between 8 and 15 percent of residents had serious problems during training [4].

So why are interviews not sufficient for assessing these characteristics, and why do some programs to use personality inventories? For one, interviewing applicants for any high-stakes position requires specific skills that interviewers are not often taught and experience they don’t often have (faculty may interview only a few candidates in a year).

Second, commonly asked questions do little to reveal whether someone is a good fit with the program or will be successful. Third, the context of the interview is far removed from the setting in which the applicant will work. While interviewing can be stressful, it does not replicate the hospital environment’s multitasking, life-threatening emergencies, conflict, and fatigue. As a consequence, the traditional interview process provides little more than an opportunity for candidates to put on their best face and answer what they think the interviewer wants to hear and for the program to sell itself.

Aware of the large investment made in new hires, the business world has studied performance prediction for years. Despite the popularity of personality inventories, their validity remains controversial [5, 6].What has gained traction with a fairly robust evidence base is the behavioral-based interview (BBI) technique, grounded in the logic that past behavior predicts future behavior. Described by Janz in 1982, the BBI seeks to discover how the candidate acted in specific employment-based situations focusing on experiences, behaviors, skills and abilities that are job-related [7]. As opposed to asking hypothetical questions (e.g., “what would you do if…”), which allow candidates to provide the response they think is expected, the interviewer asks the applicant to discuss an actual experience and how he or she responded to it. For example, in looking at a candidate’s ability to manage conflict, he or she may be asked to “give an example of a situation in which you had difficulties or conflict with a team member and how you resolved it.” To explore a candidate’s ability to respond to negative feedback, he or she may be asked to “tell about a time when you were criticized for your performance and how you responded.” Further details of the behavior and outcome are elicited to facilitate understanding about the respondent’s motivation and the final outcome. Situations discussed may not be medically related but utilize relevant skills.

Interpreting and assessing behavioral interviews is a systematic process: the interviewer considers how relevant the skills described are to the job description, how recent the situation occurred and most importantly, what the applicant’s reasoning was, and what he or she learned from that experience [8]. An applicant’s description of what, in hindsight, he or she would have done differently may provide better insight into a candidate’s commitment to reflection and self-improvement.

While supportive research exists in the nonmedical literature on the reliability and validity of the BBI [7], there is a paucity of medical education literature on the subject. Only two studies look at the BBI in residency recruitment. One radiology program looked at the predictive ability of behavioral interviewing by comparing scores to program director’s assessments 4 years later, showing predictive utility for conscientiousness and interpersonal skills [9]. An anesthesia program looked at how this approach was received and found that the BBI process was acceptable to both the interviewers and the candidates [10].

Is behavioral interviewing going to match the perfect candidate with the perfect program every time? There is certainly no evidence to support that outcome at this time. Can it provide a better understanding of a candidate and make for a more interesting experience for both interviewer and candidate? Anecdotally, the experience of this writer and others says yes. Conversations that encourage thoughtful dialogue are much more likely to conclude with both participants more fully appreciating what the other has to offer. There is more to learn from discussing a specific experience that led to growth than from questions about why a candidate chose a given specialty.

Interviews are an important part of the application process. In the most recent survey by the National Residency Matching Program, program directors reported that an applicant’s interaction with faculty during the visit and interpersonal skills were among the top three factors contributing to his or her ranking [11]. A behaviorally based interview or personality test is not likely to undermine a candidate’s chance of matching. What is probably most important for a candidate to do on a visit to a program is to treat everyone (particularly program staff) with courtesy and respect. Program directors readily acknowledge that, each year, candidates are taken out of the pool simply for their poor treatment of staff during their visit. The feeling is if someone can’t behave respectfully during the one-day visit, how will he or she be over the next 3 to 5 years? Finally, showing genuine enthusiasm and interest will convince most interviewers that you could be a good fit in their program.



References

  1. Hamdy H, Prasad K, Anderson MB, et al. BEME systemic review: predictive values of measurements obtained in medical schools and future performance in medical practice. Med Teach. 2006;28(2):103-116.
  2. Greenburg DL, Durning SJ, Cohen DL, Cruess D, Jackson JL. Identifying medical students likely to exhibit poor professionalism and knowledge during internship. J Gen Intern Med. 2007;22(12):1711-1717.
  3. Yao DC, Wright SM. National survey of internal medicine residency program directors regarding problem residents. JAMA. 2000;284(9):1099-1104.
  4. The Problem Resident [video]. Philadelphia, PA: The American Board of Internal Medicine; 1992.
  5. Murphy KR. Why don’t measures of broad dimensions of personality perform better as predictors of job performance? Hum Performance. 2005;18(4):343-357.
  6. McDaniel MA, Whetzel D, Schmidt FL, Maureer SD. The validity of employment interviews: a comprehensive review and meta-analysis. J Applied Psychol.1994;79(4):599-616.
  7. Janz T, Hellervik L, Gilmore D. Behavior Description Interviewing. Newton, MA: Allyn and Bacon Publishers; 1986.
  8. Byham W, Krauzer S. The Selection Solution: Solving the Mystery of Matching People to Jobs. Pittsburgh: DDI Press; 1996.
  9. Altmaier EM, Smith WL, O’Hallaran CM, Franken EA Jr. The predictive ability of behavioral-based interviewing compared with traditional interviewing in the selection of radiology residents. Investigative Radiol. 1992;27(5):385-389.
  10. Easdown J, Castro PL, Shinkle EP, Small L, Algren J. The behavioral interview, a method to evaluate ACGME competencies in residents. http://www.siumed.edu/resaffairs/pdfs/The%20Behavioral% 20Interview%20Method.pdf. Accessed November 13, 2012.
  11. Signer M. Presentation at: Association of Program Directors in Internal Medicine National Meeting, October 12, 2012; Phoenix, AZ. http://www.im.org/Meetings/CurrentMeetings/AIMW12/Presentations /Documents/2012%20APDIM%20Fall%20Meeting/APDIM%20PS%20I_Signer.pdf. Accessed November 13, 2012.

Asher Tulsky, MD, is an associate professor of medicine and associate program director at the University of Pittsburgh Internal Medicine Residency program in Pennsylvania. He has been actively involved in residency education for more than 15 years, with a focus on evaluation, advising, and intern recruitment.

Commentary 2

by Julie M. Aultman, PhD, and Matthew J. Zirwas, MD

There are two big academic hurdles for those who want to become physicians: being admitted into medical school and matching into a residency program. Those who make it over these hurdles shape the future character of medicine as a profession. Who gets admitted into medical school and residency, then, is of crucial importance, obviously, to both those seeking admission and those who are interested in how medicine performs.

The admission processes of medical schools and residencies are not devoid of ethical considerations. Requiring applicants to take personality tests to predict future performance, for example, may introduce unfair disadvantages for no justifiable reason. The use of personality tests in the admission process will be examined from two perspectives in the attempt to answer the fundamental ethical questions:

  1. How should the interests of each different stakeholder (applicants, medical schools and residency programs, society) be identified, considered, and incorporated into the admissions process?
  2. If the interests of the different stakeholders conflict with each other, how should these conflicts be resolved?

Identifying the Interests of Stakeholders

We will begin by considering the instrumental and intrinsic interests of each set of stakeholders in the system. Intrinsic interests are those interests that are valued not because they lead to something else, but for their own sake, e.g., human flourishing and happiness. Instrumental interests are those interests that may lead to something else that is good or valued, e.g., earning a high income. Some things, such as education, may be intrinsic interests to some, but extrinsic interests to others.

Applicants. The primary interest of applicants is to get a medical education, which may lead to intrinsic interests, e.g., human flourishing. Thus they have an interest in a process that is fair and consistent (that will treat applicants equally so as not put any at an unfair disadvantage), transparent and predictable (that will specify what is desired in advance so candidates can work to become competitive), and based on variables over which they have some measure of influence (that focus on achievement—what one does—rather than what one is).

Variables that could be fairly, consistently, and transparently assessed, and which applicants can influence to some degree include academic performance, volunteer experiences, and research experiences. Variables over which applicants do not have control, or which may not be transparently and consistently used in the admissions process, include, for example, ethnicity, personality characteristics, and family background (e.g., parents who are physicians). Judgment on such variables can also be considered unfair or discriminatory because they are unrelated to a person’s qualifications for a position (e.g., gender, race, sexual orientation) or because doing so undermines equality of opportunity. The personality test given to Brent may be fair if it is administered to all applicants, but it is not transparent—he was unaware in advance of having to take it—or based on achievement.

Medical education institutions. Medical schools and residency programs (medical institutions) all have a common instrumental interest: to select applicants who will successfully complete their training and become licensed physicians who will keep up on new trends, information, technologies, and practice in clinical medicine or research. Beyond that, each medical institution has its own interest in producing a certain type of graduate, in part to contribute to satisfying society’s interests. Examples of the types of graduates a medical institution may wish to produce include: primary care physicians, leaders in academic medicine (chairs and deans), researchers, excellent clinicians, leaders in medical administration, leaders in organized medicine, or physicians who practice in underserved locations.

Society. Society’s interest in the admissions process is to have those applicants selected who will become physicians who meet the needs of society either by providing high-quality, cost-effective medical care or by promoting medical advancements through research, efficient administration, or entrepreneurial activities. This ultimately leads to human flourishing by preserving life and alleviating suffering. Fundamental to this interest is a secondary interest in the types of physicians who meet the specific health care needs of society both in terms of specialty and geographic distribution (the most acute needs at present are in primary care and underserved areas, both rural and inner-city).

It turns out that the interests of society and medical schools are relatively similar: to select applicants who will become physicians who may fill the necessary roles in medicine. We can consider these two stakeholder groups to have compatible interests specific to the admission process, although they may have diverging subsequent instrumental and intrinsic interests.

The primary difference is that there may not be alignment between the roles medical institutions emphasize and value in their selection and the roles society may value. The presumption is that medical institutions are more interested in admitting applicants who will contribute to the reputation (and thus future success) of the institution, while society’s interests focus on ensuring there are an adequate number of qualified physicians to meet the patient care needs of the population. In essence, though, both medical institutions and society have an interest in an admission process that selects applicants based on predicting what type of physicians they are likely to become.

Are Personality Tests Accurate and Useful Predictors of Performance?

Personality traits have been shown to be relatively stable over time, starting at a young age [1], and a recent review of the existing literature concluded that personality assessments that identify conscientiousness, agreeableness, extraversion, openness, and emotional stability (known as the “big five”) are effective predictors of performance in medical school (assuming a minimum level of academic ability has been demonstrated) [2, 3]. Personality tests, however, may not be reliable predictors of academic performance, especially if those administering the tests fail to consider the variables that can skew results (e.g., dishonest answers, a poor environment with multiple interruptions, mood). Moreover, there is little data on how well personality tests, grades, MCAT scores, or other variables predict a physician’s performance or happiness over the course of his or her career. The lack of data is due in part to the fact that there is no metric for performance as a physician. More work is needed to determine whether variables such as personality characteristics can predict future performance or happiness.

Resolving the Conflict

As previously stated, applicants desire an admissions process in which they have a fair chance of success, while society and medical institutions desire a process that will predict the type of physician an applicant is likely to become. Is there a way to achieve compatibility among all stakeholders, including applicants such as Brent?

Disclosing the use of personality tests in advance may be a way to fulfill society’s and medical schools’ interest in assessing applicants’ personalities and applicants’ interest in transparency. Of course, this may make the results of personality assessments more subject to student intervention and less accurate. Applicants may answer questions based on what they think the institution desires rather than what they are actually thinking or feeling, even preparing themselves to take these tests just so they can “pass.” This may make personality assessments more transparent and therefore more acceptable to some applicants, but perhaps less useful or appealing to schools.

Ethical Guidance

Ramifications for the individual student. From the point of view of an ethicist, personality tests should be introduced into the admissions process with caution. Even if such tests are used in a clear, transparent, equal way, personality is no more under the control of applicants than are variables such as height, eye color, or ethnicity. Is it fair or unfair to select students based on these uncontrollable variables? If personality attributes are fixed core determinants of physicians’ success analogous to intelligence, and we consider it fair to disqualify applicants on the basis of their intellectual performance, it would be fair to consider personality in a similar way.

On the other hand, the analog of a personality test is an IQ test—it measures capacity, rather than achievement. Achievements should be the residency interviewer’s primary focus because they show what the student can accomplish, not merely what attributes he or she possesses. Furthermore, the student’s academic credentials are both more reliable predictors of performance in medical school and more reliably measurable. The question is less whether using IQ or personality tests is fair than whether using them is as effective as assessing performance.

In looking at the case of Brent, for example, the residency interviewer should focus on the work he’s done, while being transparent about the type of physician the institution and community is looking for. If Brent meets the minimum criteria for achievements, such as academic performance and engagement, research, and extracurricular and volunteer activities, and if he professes that the type of physician the institution seeks to produce is in alignment with his goals, then a personality assessment can be used as a secondary instrument to determine Brent’s fit with the environment of the institution and, potentially, as one of several tools to guide professional development.

Frequent personality testing prior to pursuing medicine—before applying to medical school or committing to a premed major—could help counsel students regarding whether or not they would be happy as physicians (or a particular type of physician). Such testing prior to medical school and residency would also alleviate concerns among applicants about the legitimacy of the tests, since personality testing over time would reveal more consistent data with fewer errors and make clearer the intrinsic and extrinsic interests possessed by applicants.

To ensure a fair, transparent admissions process that would not be discounted or resisted by applicants, the purpose, risks (false-negative and false-positive rates), and quality (reliability, validation, freedom from bias) of these assessments should be disclosed in advance to all applicants (e.g., on an institution’s web page or list of criteria for residency matching). Through such disclosure, admissions committees may put candidates at ease, possibly even persuading them to answer honestly.

Implications for society. There is a broader consideration at play here as well. The personalities of individual medical students can, collectively, shape the character of the profession as a whole. If medical schools and residency programs preferentially admit applicants who are compassionate, empathic, and intrinsically motivated (dubbed “type I” people in the 2009 book Drive, by Daniel Pink [4], their actions will lead to a far different profession than selecting those who are organized, achievement-oriented, and extrinsically motivated (“type X”).

A medical field dominated by type I doctors would believe that the intrinsic satisfaction of using one’s natural talents and skills to help patients was the primary reward of being a physician. Extrinsic factors like hours worked, compensation, and external recognition would still matter, but would be secondary. A medical field composed mostly of type X individuals would, while acknowledging the importance of intrinsic factors, focus primarily on the extrinsic factors.

If one considers those with similar natural talents, type I individuals will almost always outperform type X individuals over the long run (decades) but not in the short run (years), especially if the performance metrics are well defined and easily measured. Type X students who desire to become physicians would be expected to outperform their type I classmates in the education years, when outcome variables (e.g., grades, standardized test scores) are well defined but to underperform in the decades of medical practice that follow, when outcome variables (e.g., being a “good doctor”) are not well defined.

Accepting Pink’s categories would behoove us to identify and promote type I candidates for medical education. On the other hand, it can also be argued that a diverse range of personalities would best serve the pluralistic values of society and its interests, which may not be consistent with the interests of a homogenous profession. Either goal—creating a primarily type-I or a psychologically diverse physician workforce—could be furthered by accurate and reliable personality testing.


References

  1. Costa PT Jr, Herbst JH, McCrae RR, Siegler IC. Personality at midlife: stability, intrinsic maturation, and response to life events. Assessment. 2000;7(4):365-378.
  2. McCrae RR, John OP. An introduction to the five-factor model and its applications. J Pers. 1992;60(2):175-215.
  3. Doherty EM, Nugent E. Personality factors and medical training: a review of the literature. Med Educ. 2011;45(2):132-140.
  4. Pink D. Drive: The Surprising Truth About What Motivates Us. New York, NY: Riverhead Books; 2009.

Julie M. Aultman, PhD, philosopher and bioethicist by training, is an associate professor of family and community medicine, director of the Bioethics Certificate Program for health care professionals and students, and chair of the institutional review board at Northeast Ohio Medical University in Rootstown. She also directs and coordinates the bioethics curriculum at Northeastern Ohio Universities College of Medicine, which is expected to evolve into a master’s degree program in the next few years.

Matthew J. Zirwas, MD, is an associate professor of dermatology at The Ohio State University College of Medicine in Columbus. He is the residency program director in dermatology and has worked on several curriculum redesign and admissions committees. His primary nonclinical research interest is resident and medical student selection, specifically the identification of effective predictors of performance during medical school and residency.

Assessing Noncognitive Attributes: The Primary Care Orientation Scale, December 2012

Bias in Assessment of Noncognitive Attributes, December 2012

Becoming a Doctor in Europe: Objective Selection Systems, December 2012

The people and events in this case are fictional. Resemblance to real events or to names of people, living or dead, is entirely coincidental. The viewpoints expressed on this site are those of the authors and do not necessarily reflect the views and policies of the AMA.