In 2012, the retail store Target inadvertently informed a father that his young daughter was pregnant.¹ The few who knew this before him were the daughter, her mother, and Target’s new artificial intelligence (AI) algorithm. After half a century of failed starts, the AI revolution had begun in earnest. The genie was effectively out of the bottle.

In statements that echo the early era of the internet, many businesses conflate adoption of AI today with survival tomorrow.^2,3 This sentiment, or warning, is the same for emergency physicians and the emergency department (ED) ecosystem. Dismiss AI’s future impact on your practice and risk the same as those who once dismissed the internet’s potential impact on health care.

Sentient Robots?

Don’t be fooled by its name. AI programs and algorithms don’t think, understand, or comprehend. At its core, AI does one thing: categorize patterns in data. Enormous matrices filled with data vectors are ingested and predictions made that either classify what group a new data vector might belong to or how much that vector might influence the outcome you’re interested in, based on deciphered data patterns.⁴ “Self-driving” algorithms merely ingest a matrix of image pixel data vectors, match it to the most probable known pattern, and output the action for that pattern (e.g., “apply brake pedal”). All these programs do is detect a pattern and output a prediction with the highest probability. Yet, the following examples show how incredible the results can be.

Using AI, programs have been created that detect cargo containers containing smuggled goods, assess a city for hurricane damage in a few hours instead of a few days, and predict when a police officer’s radio channel should be activated seconds before the 9-1-1 call is made. With such huge successes comes the confidence to grasp for the holy grail. Whether the aspiration is to save lives, save money, or grab a slice of one the most lucrative business sectors, AI experts have turned their focus on health care.

AI in Health Care

There are still big issues with AI in health care. First, while 70 percent accuracy might be career changing for a marketing executive, anything less than 99 percent fails to meet any but the most relaxed clinician’s standards.⁵ Second, we’ve heard horrible stories of bigotry and ethnic marginalization by AI programs.⁶ Finally, the inaccuracy of these programs can be ridiculously obvious despite high algorithm confidence.⁷ During COVID, thousands of clinical algorithms were created and nearly all of them failed. One of the greatest cited reasons for failure was unbalanced teams with either too few clinicians or too few AI experts, or both. Although there is growing recognition of the need for all stakeholders (including the physician users) to audit these programs at every stage, inclusion of practicing clinicians during AI creation remains uncommon outside of advisory boards with very limited influence.

Unfortunately, there are some ugly counterpoints to our disapproval with which we must contend. Different clinicians managing the same patient often make very different decisions, a phenomenon known as interpractitioner variability. However, variability even exists within a single practice, with some physicians making decisions depending on the time left in their shift, fatigue from consecutive shifts worked, the recipient of their sign-out, whether they recently heard about a nasty medical malpractice lawsuit, or any number of things that bear absolutely no influence on the actual probability of disease. This is not to mention well-known examples of gender and ethnic discrimination by clinicians. Sadly, our own decision-making is also inherently flawed. In fact, one might argue it’s far easier to program bias out of an algorithm than out of a human being.

Interpractitioner variability is one of the greatest inspirations pushing AI into the medical decision space. The companies paying for those decisions are especially incentivized. Although efficient medical practice is noble, cost-cutting at the expense of accurate diagnoses or appropriate disposition must be avoided.

When the emergency physician follows an incorrect AI recommendation, we all know who’s liable. We might try to defend ourselves by saying the ECG interpretation said “normal” but forgiveness for the clinician quickly wanes when the outcome is untoward. Alternatively, choosing to disagree with the computer’s interpretation of “acute STEMI” can be even harder to defend when the physician proves to be wrong.

At least with an ECG interpretation we can typically see what’s driving the interpretation and document our disagreement, but most algorithms are opaque in their operation. Deep learning algorithms known as neural networks are notorious for having “black box” operations where even the programmer is unsure how the computer derived the solution.⁸ Even when the mathematics applied to find the solution are readily reviewed, it can practically require a master’s degree in computer science or AI to understand it well.

Several programs have been created, deployed, and even sold to other companies with no supporting published studies. Many deserve more inspection because even when lab testing shows great results, AI algorithms can be rife with things such as correlation traps, biases, and self-fulfilling prophecies that cloak serious errors. Currently, most clinicians lack sufficient understanding of these programs and their genesis, or of how to interpret the results well enough to unveil these errors during their clinical shifts.

The incorrect response is to lean away from AI. The potential benefit of these programs is too great and the quality of predictions improving too quickly to expect we have more than six or seven years before they will be an integral part of clinical practice. But if patient-focused clinicians are not leading this era, others will. It is vastly important for experienced, practicing clinicians to be involved at all stages of algorithm development. Physicians need to ensure that recommendations are delivered in a way such that their use or dismissal is easily defensible, causation is assured with all correlations, and the results are both understandable and useable by the average clinician.

AI programs can help us practice with more confidence, less liability, less variability, and most importantly, better clinical outcomes. Our participation and open discussion of their inner workings is what will make the difference.

Dr. Pangia serves as a regional medical director for emergency, hospitalist, and telemedicine as well as the regional analytics director for TeamHealth. He is a practicing emergency physician, data scientist, and a Master of Science candidate in artificial intelligence at Johns Hopkins University. He’s been engaged in pre-hospital care since 1992 and an emergency physician since 2008.

References

1. Hill, K. How Target figured out a teen girl was pregnant before her father did. Forbes. https://www.forbes.com/sites/kashmirhill/2012/02/16/how-target-figured-out-a-teen-girl-was-pregnant-before-her-father-did. Published February 16, 2012. Accessed September 22, 2022.
2. Marr, B. Why every company needs an artificial intelligence (AI) strategy for 2019. Forbes. https://www.forbes.com/sites/bernardmarr/2019/03/21/why-every-company-needs-an-artificial-intelligence-ai-strategy-for-2019. Published March 21, 2019. Accessed September 22, 2022.
3. Quantum Black AI by McKinsey. The state of AI in 2021: survey. https://www.mckinsey.com/businessfunctions/quantumblack/our-insights/global-survey-the-state-of-ai-in-2021. Published December 8, 2021. Accessed September 22, 2022.
4. Taulli, T. Artificial Intelligence Basics: A Non-Technical Introduction. Monrovia: Apress; 2019.
5. Tomaszewski CA, Nestler D, Shah KH, Sudhir A, Brown MD. Clinical policy: critical issues in the evaluation and management of emergency department patients with suspected non-ST-elevation acute coronary syndromes. Ann Emerg Med. 2018;72(5).
6. Buolamwini, J. Artificial intelligence has a problem with racial bias. here’s how to solve it. Time. https://time.com/5520558/artificial-intelligence-racial-gender-bias. Published February 7, 2019. Accessed September 22, 2022.
7. Neural net guesses memes. Twitter thread. https://twitter.com/ResNeXtGuesser/status/1458326818855940096?ref_src=twsrc%5Etfw. Published November 10, 2021. Accessed September 22, 2022.
8. Blazek, P. Why we will never open deep learning’s black box. https://towardsdatascience.com/why-we-will-never-open-deep-learnings-black-box-4c27cd335118. Published March 2, 2022. Accessed September 22, 2022.