Virtual Mentor. December 2012, Volume 14, Number 12: 998-1002.
State of the Art and Science
Bias in Assessment of Noncognitive Attributes
Uncritical reliance on individual physicians’ tacit knowledge about professional competence could lead to evaluating applicants and students according to idiosyncratic or outmoded standards.
Rick D. Axelson, PhD, and Kristi J. Ferguson, MSW, PhD
Professional competence for physicians, as defined by Epstein and Hundert , is:
the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served .
As implied by their definition, noncognitive traits figure prominently in Epstein and Hundert’s discussion of physicians’ professional competence. They cite attributes such as respect for patients, caring, emotional intelligence, teamwork, tolerance of ambiguity and anxiety, and basic communication skills as fundamental components of professional competence.
Although the importance of characteristics like those mentioned above is clear, it is quite difficult to assess the extent to which individuals possess them. Traits and skills related to providing humane medical care such as “caring” and “emotional intelligence” are much easier to recognize in practice than they are to explicitly define and measure. The crux of the problem is that traits and skills that develop over time through personal experience (i.e., learning-by-doing) in various social contexts can be difficult to express in words. They are often described as “tacit knowledge,” i.e., knowledge and skills that enable one to perform certain tasks without necessarily fully knowing, or being able to explain, how one does it. Lam notes that in contrast to the type of knowledge associated with cognitive skills (explicit knowledge), tacit knowledge is personal and contextual . Consequently, it is difficult to articulate, formalize, and share with others.
Although physicians’ tacit knowledge enables them to recognize similar competence in others, there are two major drawbacks to overreliance upon tacit knowledge as the basis for admission and evaluation processes. First, physicians’ tacit knowledge reflects their personal, perhaps idiosyncratic, understandings of the essential noncognitive traits and skills. These views may vary considerably among faculty. To be used effectively in admission and evaluation processes, these views would need to be synthesized and articulated as a shared vision of the medical school community. Secondly, because tacit knowledge develops through social interaction over time, it most likely contains outmoded beliefs and biases that hamper objective evaluation of others. A recent AAMC literature review described evidence of erroneous tacit knowledge in the form of unconscious gender or race/ethnicity bias . The review cited several studies showing that evaluators’ awareness of gender or race/ethnicity caused them to mistakenly favor one equally qualified candidate over another. Thus, unchecked reliance upon tacit knowledge can result in biased recruitment and evaluation decisions.
Therefore, the central challenge in evaluating noncognitive traits is to leverage the useful portions of physicians’ tacit knowledge into a common understanding of the most essential traits, while at the same time minimizing the influence of personal biases and irrelevant or mistaken information. Ultimately, if we are to select and develop physicians’ capacity for requisite noncognitive skills and traits, we need reliable, valid, and transparent methods for measuring them.
In the following section, we outline strategies for refining organizations’ processes to define and assess crucial noncognitive attributes. Effective use of research methods and data to move toward more explicit understanding of the desired characteristics and valid assessment of them is the guiding principle for this approach. The four steps are intended as elements of an iterative cycle to continuously improve processes for evaluating noncognitive attributes.
Improving Assessment of Noncognitive Attributes
Although there are numerous types of feedback that could be provided to evaluators, here we describe an analysis that provided evaluators feedback on bias in their evaluations.
In a recent study at the University of Iowa, we analyzed 5 years of clinical performance evaluation forms for evidence of unconscious gender bias in the ratings of our medical students . Our method involved examining whether the meaning of adjectives was affected by the gender of the student being rated. Within a factor analysis framework, highly intercorrelated groups of adjectives are interpreted as having a similar meaning; the common meaning for a given adjective grouping is represented by an underlying factor. If raters use the same meaning of the adjective regardless of the student’s gender, then the expected pattern of intercorrelations and underlying factors among adjectives would be the same for men and women students. This hypothesis was tested statistically using Multigroup Confirmatory Factor Analysis (CFA). (See Brown  for an accessible description of this technique.)
From this analysis, we found that raters did, in fact, interpret the adjectives (i.e., “measurement models”) differently based on the gender of the student being rated. These different measurement models resulted in gender-biased evaluations. Women were given more credit than comparable men for being “compassionate,” “sensitive,” and “enthusiastic,” and men were given more credit than comparable women for being “quick learners.” Thus, this type of analysis enabled us to raise evaluators’ awareness of an unconscious bias evident in the pattern of their ratings.
In sum, physicians’ tacit knowledge of vital noncognitive attributes provides invaluable raw data for developing, implementing, and refining assessment processes. As outlined in the steps above, qualitative and quantitative research methods can facilitate efforts to externalize tacit knowledge, improve measurement processes, and correct implicit biases in judgments based upon tacit knowledge. Ultimately, however, it is physicians’ reflective and judicious use of such research that will enable them to create increasingly meaningful and accurate processes for assessing noncognitive attributes.
Rick D. Axelson, PhD, is an assistant professor in the Department of Family Medicine and a program evaluation consultant for the office of consultation and research in medical education at the University of Iowa Carver College of Medicine in Iowa City. Dr. Axelson has developed and directed academic program assessment and institutional research offices at the University of Missouri-Kansas City, the University of South Alabama, and Riverside Community College. His research interests include program evaluation (theory, methods, and practice), learning outcomes assessment, and the development of practical methods for assessing the social, cognitive, and psychological factors affecting students’ engagement in learning activities and environments.
Kristi J. Ferguson, MSW, PhD, is a professor of general internal medicine, director of the office of consultation and research in medical education, and director of the master’s in medical education program at the University of Iowa Carver College of Medicine in Iowa City. Her research interests include assessing the validity of measures of student performance, assessing the predictive value of a small group experience during the admissions process, and evaluating students’ ability to recognize team behaviors in a simulation environment.
Related in VM
The viewpoints expressed on this site are those of the authors and do not necessarily reflect the views and policies of the AMA.
© 2012 American Medical Association. All Rights Reserved.