In hospitals across the United States, AI-powered models are already improving hospital workflows, helping emergency room staff triage patients and predict mortality risk, and guiding diagnostic and treatment decisions in the clinic – all without any system in place to make sure these uses are safe and equitable, says Carole Federico, a GSK.ai-Stanford Ethics fellow at Stanford University.
That need not be the case, she says: “There are ways to start implementing changes today, and to do it intelligently.”
Federico and Artem Trotsyuk, also a GSK.ai-Stanford fellow, recently co-authored a paper with Stanford professors Mildred Cho, Russ Altman, and David Magnus titled Stronger Regulation of AI in Biomedicine, published in Science Translational Medicine. It’s an urgent plea for governmental and institutional leadership to work together on a nationwide plan to make sure AI does no harm to patients.
And, they say, there’s no need to reinvent the wheel. Indeed, the paper itemizes a series of actions for which there are precedents in other contexts. According to Trotsyuk, “These things that have been done previously just need to be translated into corollaries that can be applied to biomedical AI.”
From Blueprint to Reality, Top-Down
A recent white paper outlining an AI Bill of Rights published by the White House Office of Science and Technology Policy served as the team’s jumping off point. Intended as a guide for society, the AI Bill of Rights identifies important principles to consider in the design, use, and deployment of AI in biomedicine. “It’s great as far as it goes, but it has no teeth,” Trotsyuk says. To change that, the team’s paper pairs each of the AI Bill of Rights’ major recommendations with a precedent for how it could be implemented.
A key element: top-down coordination. As an example, they point to the 1986 Coordinated Framework for the Regulation of Biotechnology, which worked with several federal agencies (U.S. Food and Drug Administration, Environmental Protection Agency, and U.S. Department of Agriculture) to develop a comprehensive approach for ensuring the safety of biotechnology products.
A similar framework for biomedical AI would involve an even more distributed network of regulatory agencies and institutions that would need to work together, Federico says. So, for example, the strategies developed by a cross-disciplinary biomedical AI expert panel would provide top-down, coordinated guidance about how biomedical AI algorithms should be evaluated by the FDA; reimbursed by Centers for Medicare and Medicaid Services; reviewed by hospital and university institutional review boards; and regulated by states, which have control over what hospitals and doctors can and cannot do. The result, Federico hopes, would be a comprehensive nationwide approach to keeping AI safe and equitable.
Guidance for Research and Clinical Care
When it comes to research, Federico and Trotsyuk enumerate several options for immediate action.
First, they propose convening a team of cross-disciplinary experts for the oversight of biomedical AI research. It would be modeled on the team of experts that coordinated research into the virulence and transmissibility of the COVID virus during the pandemic. In addition, they propose setting acceptable risk levels for federally funded research similar to those proposed by the European Union’s Artificial Intelligence Act, which could include banning certain types of research altogether; establishing a special review process for federal grants in biomedical AI that would be modeled on the Embryonic Stem Cell Research Oversight Committee; and requiring that investigators think through the impact of every research project before institutional funds are released to a grantee – a proposal modeled on Stanford’s Ethics and Society Review.
There’s also a need for post-approval oversight of AI tools that can become unsafe or biased over time as they are fine-tuned with new data. “There has to be a way of ensuring that fine-tuning is happening, and that someone is looking at how the model performs after it is fine-tuned,” Federico says. To address this problem, she and Trotsyuk point to guidelines the FDA developed for the use of machine learning in medical devices. But these guidelines don’t go far enough and need to be more specific and mandatory, they say.
The team also urges the FDA to require greater transparency from the biomedical AI designers who come to them for product approval. Specifically, Trotsyuk says, the FDA should require clarity about where the training data came from, whether it is representative of the people it will affect, and whether there’s bias in the dataset. There should also be clarity about how the algorithm is making decisions and how that will be explained to patients.
Healthcare reimbursement rules set by the Centers for Medicare and Medicaid Services can also provide both carrots and sticks for developers to ensure their products are safe, effective, and equitable. The agency can also make sure that all patients – not just those with private insurance – can share in the benefits of AI technologies.
When it comes to clinical care, there should be policies about when it’s appropriate to implement AI in a hospital system, what needs to be disclosed to patients about the use of AI in their care, and how patients can, if they wish, opt out of the use of AI in their care, Federico says. Stanford Hospital is already setting an example by developing these standards and policies in house, Trotsyuk says. And The Joint Commission, the healthcare standards-setting body that is responsible for the accreditation of hospitals, could take Stanford’s approach as a starting point for developing policies other hospitals could follow.
Immediate Action Needed
Hospitals and academic institutions are already worried that they could be implementing AI systems without knowing if they are safe and equitable, Federico says. This gives her hope that institutional-level changes, such as adoption of Stanford’s Ethics and Society Review process, could be rapidly and widely implemented.
Meanwhile, it could take some time to put in place some of the top-down regulations. But developers of biomedical AI would also benefit from greater clarity about standards for doing no harm. Without clear guardrails, they could inadvertently cause harm to patients and potentially face liability for their actions.
There need to be both carrots and sticks at all levels, Federico says. “There should be a motivation to do what’s necessary, but also penalties should developers, hospitals, or physicians not follow the rules.”
Stanford HAI’s mission is to advance AI research, education, policy and practice to improve the human condition. Learn more.