Medicine and Society
Aug 2022
Peer-Reviewed

Should We Rely on AI to Help Avoid Bias in Patient Selection for Major Surgery?

Charles E. Binkley, MD, David S. Kemp, JD, and Brandi Braud Scully, MD, MS
AMA J Ethics. 2022;24(8):E773-780. doi: 10.1001/amajethics.2022.773.

Abstract

Many regard iatrogenic injuries as consequences of diagnosis or intervention actions. But inaction—not offering indicated major surgery—can also result in iatrogenic injury. This article explores some surgeons’ overestimations of operative risk based on patients’ race and socioeconomic status as unduly influential in their decisions about whether to perform major cancer or cardiac surgery on some patients with appropriate clinical indications. This article also considers artificial intelligence and machine learning-based clinical decision support systems that might offer more accurate, individualized risk assessment that could make patient selection processes more equitable, thereby mitigating racial and ethnic inequity in cancer and cardiac disease.

Risk Assessment and Inequity

It is well documented that Black patients die more often from cancer and heart disease than do similarly matched White patients.1,2,3,4 While multiple factors account for this disparity, given equivalent indications, Black patients are less likely to receive complex cardiac and oncologic surgical treatment than White patients.5,6,7,8,9,10,11,12 This disparity has largely been attributed to lack of access to complex surgical care and patient refusal to undergo surgery.5,6,7,8,10,11,12,13,14 However, these factors disregard the role surgeons play in patient selection for major surgery and the potential for biased assessments based on race or socioeconomic status to influence surgical judgment.8,15,16 We propose that the use of artificial intelligence and machine learning (AI/ML) for clinical decision support (CDS) can reduce bias and promote data-driven decisions about patients’ eligibility for major surgery.

Patient Selection

Patient selection for major surgery is a highly venerated and rarely challenged prerogative of the surgeon.17,18,19 Surgical judgment is influenced by both objective and subjective assessments, the latter often dominating the final decision. For many types of cancer and cardiac diseases, surgery represents a patient’s only possibility for long-term survival.5,6,7,8,9,11,12,20,21 Thus, when patients are not offered surgical treatment, they are likely to die from the underlying disease. For both cardiac disease and cancer, the failure of surgeons to offer potentially lifesaving surgery likely contributes to observed racial disparities.

One of the most common reasons surgeons give when refusing to operate on a patient with an appropriate indication is that the patient is considered to be at too high a risk for complications or death.18 Surgeons assess the risks and benefits of operating and of not operating on a patient.18 Professional responsibility requires that the benefit of operating and the risk of not operating be sufficiently skewed so as to justify performing the operation.22,23 This consideration raises 2 important questions: (1) how a patient’s risk is assessed and (2) whether concern about outcome metrics unduly affects surgical judgment.

Risk assessment. Surgical risk is an assessment of the likelihood that a patient will suffer a complication or death related to an operation.18 Surgical risk calculators have been developed to predict the likelihood of perioperative morbidity and mortality.24,25,26,27 In general, as patients amass comorbidities, their surgical risk increases. Recently, frailty scores have been introduced as a way to quasi-quantitatively assess what many surgeons call the “eyeball test,” their subjective appraisal of how frail a patient is.28,29 The more frail the patient, the greater is the risk of complications and death.30 But any assessment that relies on individual observation risks introducing bias.

Indeed, surgeons have estimated similar comorbidities to be more severe in Black patients than in White patients, and Black patients have been offered aggressive treatment less often than White patients with equivalent indications.15,20,31,32,33,34,35,36 Because of the association between race and socioeconomic status, a surgeon might assess angina in a well-dressed, upper-middle-class White man differently than in a Black man experiencing housing insecurity with a medically equivalent condition. Similarly, chronic obstructive pulmonary disease of equal severity may look very different in a White woman who was driven to the consultation than in a Black woman who took public transportation and then walked several blocks to reach the surgeon’s office. Black patients who are poor and undereducated may appear on an eyeball test to be more frail and higher risk than well-nourished, well-rested patients.

Besides being biased by socioeconomic factors that may cause Black patients to be judged at higher risk for surgery, surgical decisions may also be affected by ostensibly objective data indicating that, for major cancer and cardiac surgery, Black patients have higher mortality rates, higher rates of postoperative complications, higher readmission rates, and longer lengths of stay than similarly matched White patients.8,9,14,37,38,39,40,41,42 These reported outcomes, which are closely associated with socioeconomic status, could further justify the surgeon’s subjective assessment of the patient’s potential for a successful postoperative and posthospital recovery.

Any assessment that relies on individual observation risks introducing bias. 

Elevated mortality rates of Black patients undergoing major surgery are often attributed to these patients’ lack of access to high-quality surgical care.8,9,38,41,43 The typical reasoning is that Black patients often seek care at lower quality hospitals and by less experienced surgeons, and thus they suffer complications and death more frequently.38,39,41,43 This line of reasoning presumes that Black patients themselves choose lower-quality surgical care, disregarding the distinct possibility that these hospitals and surgeons may be the only ones who are willing to accept those Black patients whom higher-quality hospitals with more experienced surgeons have deemed too high risk. A plausible scenario that deserves further investigation is whether Black patients are cared for in lower quality hospitals because the surgeons at those hospitals do not judge Black patients to be as high risk as do their colleagues at higher quality medical centers.

One proposed solution to the problem of unequal access is the take the “Access Pledge,” whereby high-quality, high-volume medical centers assure equal access to all patients.44 However, Black patients who have access to high-volume hospitals can still experience bias in selection for surgery, prompting some to seek treatment where they can access unbiased surgical assessment.

Outcome metrics. In addition to a biased subjective risk assessment, outcome metrics may affect a surgeon’s objectivity in deciding whether to recommend a patient for major surgery. As a result of excessive iatrogenic injury among hospitalized patients, the late 1990s saw the introduction of quality metrics, including surgeon-specific measures of operative mortality and major complications.45,46,47,48 While the aim was to improve the quality of surgical care, these metrics could also disincentivize some surgeons from operating on patients they perceive as too high risk. Such decisions in part reflect surgeons’ own self-interest in not having their outcome metrics “look bad” before their peers and hospital administrators. A surgeon could thus decide that there is less risk and greater benefit in not operating or greater risk and less benefit in performing the operation. Furthermore, there is no system of accountability for a surgeon’s refusal to operate on a patient, regardless of the underlying reason.49

Use of AI/ML CDS

How can potential surgeon bias in patient selection for major surgery be remedied? While interventions such as race-specific feedback on treatment completion rates and the use of nurse navigators have been shown to reduce racial disparities in care for early-stage lung cancer,50 such interventions are downstream of the potentially biased clinical decisions that directly affect patient outcomes. What is needed is an objective system that can share agency with a surgeon in selecting patients for complex surgery. The use of AI/ML CDS systems holds great promise for debiasing surgical decision making.

Implementing AI/ML CDS could debias patient selection for complex surgery in 3 ways. First, the system could provide an objective, accurate, and individualized assessment of surgical risk based on information from the patient’s medical record rather than subjective appraisals.51,52 In other settings, standardizing clinical decisions and postoperative pathways has been shown to reduce racial disparities among surgical patients.33,53,54,55 Second, the system would not be affected by concern for reported outcome metrics that might otherwise bias surgical judgment. Finally, the system could track not only the patients accepted for surgery but also those declined for surgery, thus providing a mechanism for recognizing biased trends.

Although AI/ML systems have been associated with perpetuating rather than resolving bias,56 they are neither inherently biased nor essentially unethical. One way to debias AI is by carefully examining the assumptions the algorithm uses to make predictions and the data on which the system is trained. In one study, an algorithm was used to predict which patients would have the greatest future health care needs.56 The system used data from past health care expenses and assumed the data would reflect the severity of underlying illness to predict future health needs. The algorithm systematically underestimated future health care needs for Black patients because they utilized health care resources less often than did White patients, regardless of severity of underlying illness, and thus had overall lower historic health care expenses. The algorithmic assumption was wrong in that past health care expenses did not predict future health care needs.

In the same way, AI/ML surgical risk calculators could perpetuate racial bias if the algorithm assumes that operative morbidity and mortality are due entirely to underlying patient comorbidities and inherent patient risk. Nonpatient-controlled factors, such as hospital and surgeon volume, can also affect operative morbidity and mortality.57 To make accurate predictions, an algorithm would need to weigh these other factors and not assume that operative outcome is entirely patient dependent.

An AI/ML system that is trained to make predictions based on assumptions that rely on historically biased data will perpetuate those same biases. If the assumptions can be corrected, then the predictions will become more reliable.58,59 In debiasing AI/ML CDS, it is imperative to differentiate association and causation. It may be true that being Black is associated with increased morbidity and mortality and worse long-term survival after major cancer surgery, but these outcomes are not caused by being Black. For AI/ML CDS to debias patient selection for major surgery, race-associated outcomes should be assumed to be based not solely on inherent patient risk but on inequitable health care structures as well.60

Read More

References

  1. American Cancer Society. Cancer facts and figures for African Americans, 2019-2021. American Cancer Society; 2019. Accessed June 10, 2022. https://www.cancer.org/content/dam/cancer-org/research/cancer-facts-and-statistics/cancer-facts-and-figures-for-african-americans/cancer-facts-and-figures-for-african-americans-2019-2021.pdf

  2. Cancer and African Americans. Office of Minority Health, US Department of Health and Human Services. Accessed September 29, 2021. https://minorityhealth.hhs.gov/omh/browse.aspx?lvl=4&lvlid=16

  3. Heart disease and African Americans. Office of Minority Health, US Department of Health and Human Services. Updated February 11, 2021. Accessed September 29, 2021. https://minorityhealth.hhs.gov/omh/browse.aspx?lvl=4&lvlid=19

  4. Van Dyke M, Greer S, Odom E, et al. Heart disease death rates among blacks and whites aged ≥ 35 years—United States, 1968-2015. MMWR Surveill Summ. 2018;67(5):1-11.
    View Article PubMed Google Scholar
  5. Abraham A, Al-Refaie WB, Parsons HM, Dudeja V, Vickers SM, Habermann EB. Disparities in pancreas cancer care. Ann Surg Oncol. 2013;20(6):2078-2087.
    View Article PubMed Google Scholar
  6. Bach PB, Cramer LD, Warren JL, Begg CB. Racial differences in the treatment of early-stage lung cancer. N Engl J Med. 1999;341(16):1198-1205.
    View Article PubMed Google Scholar

  7. Epstein AM, Weissman JS, Schneider EC, Gatsonis C, Leape LL, Piana RN. Race and gender disparities in rates of cardiac revascularization: do they reflect appropriate use of procedures or problems in quality of care? Med Care. 2003;41(11):1240-1255.

    View Article PubMed Google Scholar
  8. Hravnak M, Ibrahim S, Kaufer A, Sonel A, Conigliaro J. Racial disparities in outcomes following coronary artery bypass grafting. J Cardiovasc Nurs. 2006;21(5):367-378.
    View Article PubMed Google Scholar
  9. Johnston FM, Yeo HL, Clark C, Stewart JH IV. Bias issues in colorectal cancer management: a review. Ann Surg Oncol. 2022;29(4):2166-2173.
    View Article PubMed Google Scholar
  10. Lin JJ, Mhango G, Wall MM, et al. Cultural factors associated with racial disparities in lung cancer care. Ann Am Thorac Soc. 2014;11(4):489-495.
    View Article PubMed Google Scholar
  11. Lutfi W, Zenati MS, Zureikat AH, Zeh HJ, Hogg ME. Health disparities impact expected treatment of pancreatic ductal adenocarcinoma nationally. Ann Surg Oncol. 2018;25(7):1860-1867.
    View Article PubMed Google Scholar
  12. Savitch SL, Grenda TR, Scott W, et al. Racial disparities in rates of surgery for esophageal cancer: a study from the National Cancer Database. J Gastrointest Surg. 2021;25(3):581-592.
    View Article PubMed Google Scholar
  13. Coffman A, Torgeson A, Lloyd S. Correlates of refusal of surgery in the treatment of non-metastatic pancreatic adenocarcinoma. Ann Surg Oncol. 2019;26(1):98-108.
    View Article PubMed Google Scholar
  14. Richardson CJ, Itua P, Duong T, Lewars J, Tiesenga F. Racial and socioeconomic disparities in congenital heart surgery: a research article. J Card Surg. 2021;36(7):2454-2457.
    View Article PubMed Google Scholar
  15. Greenberg CC, Weeks JC, Stain SC. Disparities in oncologic surgery. World J Surg. 2008;32(4):522-528.
    View Article PubMed Google Scholar
  16. van Ryn M. Research on the provider contribution to race/ethnicity disparities in medical care. Med Care. 2002;40(1)(suppl):I140-I151.
    PubMed Google Scholar
  17. Eddy DM. Clinical decision making: from theory to practice. Anatomy of a decision. JAMA. 1990;263(3):441-443.
    View Article PubMed Google Scholar
  18. Sacks GD, Dawes AJ, Ettner SL, et al. Surgeon perception of risk and benefit in the decision to operate. Ann Surg. 2016;264(6):896-903.
    View Article PubMed Google Scholar
  19. Yule S, Flin R, Paterson-Brown S, Maran N. Non-technical skills for surgeons in the operating room: a review of the literature. Surgery. 2006;139(2):140-149.
    View Article PubMed Google Scholar
  20. Cykert S, Dilworth-Anderson P, Monroe MH, et al. Factors associated with decisions to undergo surgery among patients with newly diagnosed early-stage lung cancer. JAMA. 2010;303(23):2368-2376.
    View Article PubMed Google Scholar
  21. Taioli E, Wolf AS, Camacho-Rivera M, et al. Racial disparities in esophageal cancer survival after surgery. J Surg Oncol. 2016;113(6):659-664.
    View Article PubMed Google Scholar
  22. Shuman AG, Fins JJ. A surgeon’s dilemma. Hastings Cent Rep. 2016;46(3):9-10.
    View Article PubMed Google Scholar
  23. Shuman AG. Contemplating resectability. Hastings Cent Rep. 2017;47(6):3-4.
    View Article PubMed Google Scholar

  24. Clark DE, Fitzgerald TL, Dibbins AW. Procedure-based postoperative risk prediction using NSQIP data. J Surg Res. 2018;221:322-327.

    View Article PubMed Google Scholar
  25. Cohen ME, Liu Y, Ko CY, Hall BL. An examination of American College of Surgeons NSQIP Surgical Risk Calculator accuracy. J Am Coll Surg. 2017;224(5):787-795.e1.
    View Article PubMed Google Scholar
  26. Lubitz AL, Chan E, Zarif D, et al. American College of Surgeons NSQIP risk calculator accuracy for emergent and elective colorectal operations. J Am Coll Surg. 2017;225(5):601-611.
    View Article PubMed Google Scholar
  27. Sacks GD, Dawes AJ, Ettner SL, et al. Impact of a risk calculator on risk perception and surgical decision making: a randomized trial. Ann Surg. 2016;264(6):889-895.
    View Article PubMed Google Scholar
  28. Allen KB. Frailty: it’s hard to define, but you know it when you see it. J Thorac Cardiovasc Surg. 2014;148(6):3117-3118.
    View Article PubMed Google Scholar

  29. Speir A. Defining frailty: “I know it when I see it.” J Thorac Cardiovasc Surg. 2015;149(3):875-876.

    View Article PubMed Google Scholar
  30. Sepehri A, Beggs T, Hassan A, et al. The impact of frailty on outcomes after cardiac surgery: a systematic review. J Thorac Cardiovasc Surg. 2014;148(6):3110-3117.
    View Article PubMed Google Scholar
  31. Green AR, Carney DR, Pallin DJ, et al. Implicit bias among physicians and its prediction of thrombolysis decisions for black and white patients. J Gen Intern Med. 2007;22(9):1231-1238.
    View Article PubMed Google Scholar
  32. Haider AH, Sexton J, Sriram N, et al. Association of unconscious race and social class bias with vignette-based clinical assessments by medical students. JAMA. 2011;306(9):942-951.
    View Article PubMed Google Scholar
  33. Lau BD, Haider AH, Streiff MB, et al. Eliminating health care disparities with mandatory clinical decision support: the venous thromboembolism (VTE) example. Med Care. 2015;53(1):18-24.
    View Article PubMed Google Scholar
  34. Sabin JA, Rivara FP, Greenwald AG. Physician implicit attitudes and stereotypes about race and quality of medical care. Med Care. 2008;46(7):678-685.
    View Article PubMed Google Scholar
  35. Schulman KA, Berlin JA, Harless W, et al. The effect of race and sex on physicians’ recommendations for cardiac catheterization. N Engl J Med. 1999;340(8):618-626.
    View Article PubMed Google Scholar

  36. van Ryn M, Fu SS. Paved with good intentions: do public health and human service providers contribute to racial/ethnic disparities in health? Am J Public Health. 2003;93(2):248-255.

    View Article PubMed Google Scholar
  37. Ellis L, Canchola AJ, Spiegel D, Ladabaum U, Haile R, Gomez SL. Racial and ethnic disparities in cancer survival: the contribution of tumor, sociodemographic, institutional, and neighborhood characteristics. J Clin Oncol. 2018;36(1):25-33.
    View Article PubMed Google Scholar

  38. Khera R, Vaughan-Sarrazin M, Rosenthal GE, Girotra S. Racial disparities in outcomes after cardiac surgery: the role of hospital quality. Curr Cardiol Rep. 2015;17(5):29.

    View Article PubMed Google Scholar

  39. Lam MB, Raphael K, Mehtsun WT, et al. Changes in racial disparities in mortality after cancer surgery in the US, 2007. JAMA Netw Open. 2020;3(12):e2027415.

  40. Mehtsun WT, Figueroa JF, Zheng J, Orav EJ, Jha AK. Racial disparities in surgical mortality: the gap appears to have narrowed. Health Aff (Millwood). 2017;36(6):1057-1064.
    View Article PubMed Google Scholar
  41. Rangrass G, Ghaferi AA, Dimick JB. Explaining racial disparities in outcomes after cardiac surgery: the role of hospital quality. JAMA Surg. 2014;149(3):223-227.
    View Article PubMed Google Scholar
  42. Sukumar S, Ravi P, Sood A, et al. Racial disparities in operative outcomes after major cancer surgery in the United States. World J Surg. 2015;39(3):634-643.
    View Article PubMed Google Scholar
  43. Bristow RE, Zahurak ML, Ibeanu OA. Racial disparities in ovarian cancer surgical care: a population-based analysis. Gynecol Oncol. 2011;121(2):364-368.
    View Article PubMed Google Scholar
  44. Binkley CE, Kemp DS. Ethical centralization of high-risk surgery requires racial and economic justice. Ann Surg. 2020;272(6):917-918.
    View Article PubMed Google Scholar
  45. Clark RE. It is time for a national cardiothoracic surgical data base. Ann Thorac Surg. 1989;48(6):755-756.
    View Article PubMed Google Scholar
  46. Daley J, Khuri SF, Henderson W, et al. Risk adjustment of the postoperative morbidity rate for the comparative assessment of the quality of surgical care: results of the National Veterans Affairs Surgical Risk Study. J Am Coll Surg. 1997;185(4):328-340.
    View Article PubMed Google Scholar

  47. Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. National Academy Press; 2001.

    View Article PubMed Google Scholar
  48. Khuri SF, Daley J, Henderson W, et al. Risk adjustment of the postoperative mortality rate for the comparative assessment of the quality of surgical care: results of the National Veterans Affairs Surgical Risk Study. J Am Coll Surg. 1997;185(4):315-327.
    View Article PubMed Google Scholar
  49. Anyanwu AC. The vagaries of patient selection in cardiovascular surgery. J Thorac Cardiovasc Surg. 2016;152(3):842-846.
    View Article PubMed Google Scholar
  50. Cykert S, Eng E, Walker P, et al. A system-based intervention to reduce Black-White disparities in the treatment of early stage lung cancer: a pragmatic trial at five cancer centers. Cancer Med. 2019;8(3):1095-1102.
    View Article PubMed Google Scholar
  51. Balch J, Upchurch GR Jr, Bihorac A, Loftus TJ. Bridging the artificial intelligence valley of death in surgical decision-making. Surgery. 2021;169(4):746-748.
    View Article PubMed Google Scholar
  52. Loftus TJ, Tighe PJ, Filiberto AC, et al. Artificial intelligence and surgical decision-making. JAMA Surg. 2020;155(2):148-158.
    View Article PubMed Google Scholar
  53. Hoyler MM, White RS, Tam CW. Enhanced recovery after surgery protocols may help reduce racial and socioeconomic disparities in cardiac surgery. J Cardiothorac Vasc Anesth. 2020;34(2):569-570.
    View Article PubMed Google Scholar
  54. Leeds IL, Alimi Y, Hobson DR, et al. Racial and socioeconomic differences manifest in process measure adherence for enhanced recovery after surgery pathway. Dis Colon Rectum. 2017;60(10):1092-1101.
    View Article PubMed Google Scholar
  55. Wahl TS, Goss LE, Morris MS, et al. Enhanced recovery after surgery (ERAS) eliminates racial disparities in postoperative length of stay after colorectal surgery. Ann Surg. 2018;268(6):1026-1035.
    View Article PubMed Google Scholar
  56. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447-453.
    View Article PubMed Google Scholar
  57. Birkmeyer JD, Stukel TA, Siewers AE, Goodney PP, Wennberg DE, Lucas FL. Surgeon volume and operative mortality in the United States. N Engl J Med. 2003;349(22):2117-2127.
    View Article PubMed Google Scholar
  58. Johnson-Mann CN, Loftus TJ, Bihorac A. Equity and artificial intelligence in surgical care. JAMA Surg. 2021;156(6):509-510.
    View Article PubMed Google Scholar

  59. Turner Lee N, Resnick P, Barton, G. Algorithmic bias detection and mitigation: best practices and policies to reduce consumer harms. Brookings Institution. May 22, 2019. Accessed September 29, 2021. https://www.brookings.edu/research/algorithmic-bias-detection-and-mitigation-best-practices-and-policies-to-reduce-consumer-harms/

  60. Vyas DA, Eisenstein LG, Jones DS. Hidden in plain sight—reconsidering the use of race correction in clinical algorithms. N Engl J Med. 2020;383(9):874-882.
    View Article PubMed Google Scholar

Editor's Note

Background image by Laura Kostovich.

Citation

AMA J Ethics. 2022;24(8):E773-780.

DOI

10.1001/amajethics.2022.773.

Conflict of Interest Disclosure

The author(s) had no conflicts of interest to disclose.

The viewpoints expressed in this article are those of the author(s) and do not necessarily reflect the views and policies of the AMA.