AMA Journal of Ethics^®

Isaac C, Chertoff J, Lee B, Carne M. Do students’ and authors’ genders affect evaluations? A linguistic analysis of medical student performance evaluations. Acad Med. 2011;86(1):59-66.

In their recent paper, Isaac et al. [1] illuminated an intrinsic difference in the words used to evaluate the performance and behavior of male and female medical students. The investigators used a software program to categorize and analyze the text of deans’ letters (MSPEs) for 2009 applicants to the Diagnostic Radiology Residency Program at Dartmouth-Hitchcock Medical Center. The results suggest an underlying and pervasive cultural expectation of the roles and behaviors of men and women in medical school. In their words, men were more likely to be described as “responsible excellers and eager beavers,” while women were “enthusiastic worker bees with research experience” [2]. Medical students will not be shocked by these results. We live in a world of constant evaluation and rapid turnover, where busy residents have little time to get to know us, and where evaluations often consist of stock phrases and broad comparisons.

Surprisingly, Isaac et al. found no author-student pairings in which the description significantly benefited or handicapped an applicant’s chances of being ranked. These results suggest that, although a significant difference exists between the ways men and women describe male and female students, the qualities ascribed to women do not strike selection committees as more desirable than those ascribed to men or vice versa. Additionally, Isaac et al. noted that MSPEs written by men were significantly shorter than those written by women, though this difference in length had no effect on student rankings. In sum, this study identified gender-correlated differences in the language and length of MSPEs, but little effect on whether students matched to the Dartmouth program. These findings are interesting because subtle but significant differences in word choice seem to have little effect on tangible outcomes, analogous to the way that a drug that lowers LDL by 3 points doesn’t necessarily reduce a patient’s risk of MI. Despite this, the results allow for the discussion of an underlying gender-bias paradigm to be addressed using quantitative data.

As current medical students, we feel the effect of gender expectations on a daily basis. While male and female descriptors seem to have no effect on the likelihood of being ranked, they do reflect the environment in which we are immersed. In our lives, “enthusiastic worker bee” and “excellent eager beaver” are more than just word choices on MSPEs—they’re standards to which we are held and to which we mold ourselves over the course of 4 years of medical school. By the time we decide which residencies and specialties to pursue, we have undergone several years of slow and steady pressure to fit different ideals. Thus, while MSPE author-student gender combinations may not have affected Dartmouth’s ranking decisions, it is possible that our culture of expectation and pressure influenced which students applied to which residency programs in the first place.

This crucial conclusion in Isaac et al.—that the genders of the MSPE writer and student subject did not affect ranking decisions—underscores the need to evaluate factors that influence medical students’ career decision making. The investigators posit that subtle but clearly different social messages are conveyed to male and female medical students throughout their training, and that these messages contribute to unequal representation of men and women in various specialties. For example, nearly equal numbers of men and women graduated from U.S. medical schools in 2011 overall, but three-quarters of applicants to radiology residencies were male and four-fifths of applicants to ob/gyn programs were female [3]. While socialization may occur before medical school, the findings of Isaac et al. suggest that the culture of medical school may not be immune to these constructs.

Medical students consider job satisfaction, lifestyle, influential mentors, and a positive work environment when choosing a career. With respect to lifestyle, recent analyses of U.S. medical school graduates have revealed that men and women have nearly identical rates of migration away from careers with “uncontrollable lifestyle,” such as internal medicine [4, 5]. In our experience early and effective mentoring has shaped our understanding of the medical specialties and how we plan to build our careers. Both men and women reported experiencing gender and sexual discrimination during residency selection, as well as in medical school courses and clerkships and outside the medical training environment (mostly men entering ob/gyn and women entering general surgery) [6]. Women report more harassment and discrimination, but men weigh such experiences more heavily in their career decisions [6]. Our first recommendation is that the medical community would benefit from further research on medical students’ evolving impressions of specialties, perhaps with nationwide longitudinal studies beginning in the first year.

It is necessary to consider the limitations of the study data: the study had only a small sample size and was confined to a competitive residency program in a competitive field. Many factors contribute to matching, and it could be that the selection committee at Dartmouth puts particular emphasis on specific qualities that are not included in the narrative of the MSPE. For example, earning honors in fourth-year radiology elective may have been valued more highly than the subjective impressions of students derived from their MSPEs. Such a restricted selection criterion would lessen the effect of gender differences in MSPEs on whether students were eventually ranked. Thus, it would be useful to repeat this study with an eye to less competitive programs and a broad range of fields. Our second recommendation is that a larger-scale study be conducted that includes more specialties and has a larger sample size with greater power to detect differences in the proportion of ranked candidates.

If subsequent studies do identify such differences, our final recommendation is to develop a standardized protocol to help residency directors avoid being influenced by gender bias. One option would be to develop a subsection of the MSPE in which deans rate student character traits in a gender-neutral way. Isaac et al. suggested, for example, that women “showing enthusiasm” and men “taking initiative” probably represent the same character trait. These could be combined into a category like “go-getter: rate this student’s enthusiasm and initiative.” This would allow residency directors to read the nuance of the MSPE through the lens of a numerical character profile that helps to neutralize the effect of gender before the letter leaves the hands of the administrative author. A second option would be to use a software program similar to the language analysis software described in Isaac et al. to assign similar numerical values to MSPEs. Either of these approaches would provide additional standardization that could reduce the effect of gender bias in the success rate of applicants.

It is important that the medical community foster a culture of gender equality while appreciating the tendency of evaluators to choose their language according to the gender of the student being evaluated. We hope future research will elucidate factors that influence how medical students make career decisions, and if necessary, help to further standardize the evaluation and recommendation process.