AI in healthcare: paper calls for patient-first regulation  

A new paper discusses the need for federal regulation and evaluation of AI tools used in healthcare

Image: macrovector | Freepik 

The question about whether the latest and greatest artificial intelligence (AI) tool you read about in the morning paper will save your life can be difficult to answer. A new study published in the Journal of the American Medical Association (Jama) led by doctor of philosophy (PhD) John W Ayers of the Qualcomm Institute within the University of California, San Diego, finds that AI products in healthcare do not universally undergo any externally evaluated approval process assessing how they might benefit patient outcomes before coming to market.

The research team evaluated the recent White House Executive Order that instructed the Health and Human Services departments to develop new AI-specific regulatory strategies addressing equity, safety, privacy, and quality for AI in healthcare before April 27, 2024. However, team members were surprised to find the order did not once mention patient outcomes, the standard metric by which healthcare products are judged before being allowed to access the healthcare marketplace. 

"The goal of medicine is to save lives; AI tools should prove clinically significant improvements in patient outcomes before they are widely adopted," said doctor of medicine Davey Smith, head of the Division of Infectious Disease and Global Public Health at the University of California San Diego School of Medicine, co-director of the university's Altman Clinical and Translational Research Institute, and the study’s senior author.

According to the team, AI-powered early warning systems for sepsis, a fatal acute illness among hospitalised patients that affects 1.7 million Americans each year, demonstrate the consequences of inadequate prioritisation of patient outcomes in regulations. A third-party evaluation of the most widely adopted AI sepsis prediction model revealed that the system did not identify 67 per cent of patients who developed sepsis. Would hospital administrators have chosen this sepsis prediction system if trials assessing patient outcomes data were mandated, the team questioned, considering the array of available early warning systems for sepsis?

"We are calling for a revision to the White House Executive Order that prioritises patient outcomes when regulating AI products. Similar to pharmaceutical products, AI tools that affect patient care should be evaluated by federal agencies for how they improve patients' feelings, function, and survival," said Ayers. Ayers is also a deputy director of informatics at the Altman Clinical and Translational Research Institute, in addition to his Qualcomm Institute affiliation.

The team points to its 2023 study in Jama Internal Medicine on using AI-powered chatbots to respond to patient messages as an example of what patient outcome-centric regulations can achieve. “Theoretically, a study comparing standard care versus standard care enhanced by AI conversational agents might find differences in downstream care utilisation in some patient populations, such as heart failure patients. However, studies like this do not just happen unless regulators appropriately incentivize them. With a patient outcomes-centric approach, AI for patient messaging and all other clinical applications can truly enhance people’s lives,” said Nimit Desai, a research affiliate at the Qualcomm Institute, UC San Diego School of Medicine student, and study co-author. 

The team recognises its proposed regulatory strategy can be a significant lift for AI and healthcare industry partners and may not be necessary for every flavour of AI use case in healthcare. However, the researchers say excluding patient outcomes-centric rules in the White House Executive Order is a serious omission. 

The piece Regulate Artificial Intelligence in Healthcare by Prioritising Patient Outcomes was published by Jama.