Stanford
University
  • Stanford Home
  • Maps & Directions
  • Search Stanford
  • Emergency Info
  • Terms of Use
  • Privacy
  • Copyright
  • Trademarks
  • Non-Discrimination
  • Accessibility
© Stanford University.  Stanford, California 94305.
Automated real-time assessment of intracranial hemorrhage detection AI using an ensembled monitoring model (EMM) | Stanford HAI
Navigate
  • About
  • Events
  • AI Glossary
  • Careers
  • Search
Participate
  • Get Involved
  • Support HAI
  • Contact Us
Skip to content
  • About

    • About
    • People
    • Get Involved with HAI
    • Support HAI
    • Subscribe to Email
  • Research

    • Research
    • Research Programs
    • Grants
    • Marlowe (opens in new tab)
    • Student Affinity Groups
    • Centers & Labs
    • Research Publications
    • Research Partners
  • Education

    • Education
    • Executive and Professional Education
    • Government and Policymakers
    • K-12
    • Stanford Students
  • Policy

    • Policy
    • Policy Publications
    • Policymaker Education
    • Student Opportunities
  • AI Index

    • AI Index
    • AI Index Report
    • Global Vibrancy Tool
    • People
  • News
  • Events
  • Industry
  • Centers & Labs

Stay Up To Date

Get the latest news, advances in research, policy work, and education program updates from HAI in your inbox weekly.

Sign Up For Latest News

research

Automated real-time assessment of intracranial hemorrhage detection AI using an ensembled monitoring model (EMM)

Date
October 13, 2025
Topics
Healthcare
Regulation, Policy, Governance
Your browser does not support the video tag.
Read Paper
abstract

Artificial intelligence (AI) tools for radiology are commonly unmonitored once deployed. The lack of real-time case-by-case assessments of AI prediction confidence requires users to independently distinguish between trustworthy and unreliable AI predictions, which increases cognitive burden, reduces productivity, and potentially leads to misdiagnoses. To address these challenges, we introduce Ensembled Monitoring Model (EMM), a framework inspired by clinical consensus practices using multiple expert reviews. Designed specifically for black-box commercial AI products, EMM operates independently without requiring access to internal AI components or intermediate outputs, while still providing robust confidence measurements. Using intracranial hemorrhage detection as our test case on a large, diverse dataset of 2919 studies, we demonstrate that EMM can successfully categorize confidence in the AI-generated prediction, suggest appropriate actions, and help physicians recognize low confidence scenarios, ultimately reducing cognitive burden. Importantly, we provide key technical considerations and best practices for successfully translating EMM into clinical settings.

Share
Link copied to clipboard!
Authors
  • Zhongnan Fang
    Zhongnan Fang
  • Andrew Johnston
    Andrew Johnston
  • Lina Cheuy
    Lina Cheuy
  • Hye Sun Na
    Hye Sun Na
  • Magdalini Paschali
    Magdalini Paschali
  • Camila Gonzalez
    Camila Gonzalez
  • Bonnie Armstrong
    Bonnie Armstrong
  • Arogya Koirala
    Arogya Koirala
  • Derrick Laurel
    Derrick Laurel
  • Andrew Walker Campion
    Andrew Walker Campion
  • Michael Iv
    Michael Iv
  • Akshay Chaudhari
    Akshay Chaudhari
  • David B. Larson
    David B. Larson

Related Publications

The Healthcare Policies of the November 2026 Candidates
David Scheinker, Kevin Schulman, Anton Backer, Maya Shah
Quick ReadOct 20, 2026
Research
Your browser does not support the video tag.

Recent polls show that healthcare costs are a top affordability concern for American adults and that a large majority of adults, 90% of Democrats and 75% of Republicans, support a fundamental redesign or major changes to our healthcare system. Across states, political parties, and different types of office, politicians and constituents hold different views on healthcare policies and support different proposals. Various stakeholders such as patients, physicians, nurses, veterans, employers, and industry groups could potentially benefit from access to data on how their representatives’ views align with their own. There is no systematic, openly available database and analysis of these views. We sought to create the first systematic, openly available database and analysis of the healthcare policies and proposals of the leading candidates in the November 2026 elections and beyond.

Research
Your browser does not support the video tag.

The Healthcare Policies of the November 2026 Candidates

David Scheinker, Kevin Schulman, Anton Backer, Maya Shah
HealthcareRegulation, Policy, GovernanceQuick ReadOct 20

Recent polls show that healthcare costs are a top affordability concern for American adults and that a large majority of adults, 90% of Democrats and 75% of Republicans, support a fundamental redesign or major changes to our healthcare system. Across states, political parties, and different types of office, politicians and constituents hold different views on healthcare policies and support different proposals. Various stakeholders such as patients, physicians, nurses, veterans, employers, and industry groups could potentially benefit from access to data on how their representatives’ views align with their own. There is no systematic, openly available database and analysis of these views. We sought to create the first systematic, openly available database and analysis of the healthcare policies and proposals of the leading candidates in the November 2026 elections and beyond.

Roles for AI in Implementation of Medicaid Work Requirements
Michelle Mello, Himaja Nagireddy
Quick ReadSep 03, 2026
Research
Your browser does not support the video tag.

By January 2027, 44 state Medicaid programs must redetermine the eligibility of millions of beneficiaries under criteria in the 2025 budget reconciliation law (HR 1). Among the changes are “community engagement” (work) requirements and increased redetermination cadence (from annual to at least semiannual). In preparation, state Medicaid agencies have been exploring technological solutions to automate redeterminations at scale, including artificial intelligence (AI). Six agencies have decided to use AI tools so far and 21 states are considering use of them. Strategic deployment of AI can help states tackle the redetermination juggernaut, but overly broad or hasty deployment could compound the harms of HR 1.

Research
Your browser does not support the video tag.

Roles for AI in Implementation of Medicaid Work Requirements

Michelle Mello, Himaja Nagireddy
HealthcareRegulation, Policy, GovernanceQuick ReadSep 03

By January 2027, 44 state Medicaid programs must redetermine the eligibility of millions of beneficiaries under criteria in the 2025 budget reconciliation law (HR 1). Among the changes are “community engagement” (work) requirements and increased redetermination cadence (from annual to at least semiannual). In preparation, state Medicaid agencies have been exploring technological solutions to automate redeterminations at scale, including artificial intelligence (AI). Six agencies have decided to use AI tools so far and 21 states are considering use of them. Strategic deployment of AI can help states tackle the redetermination juggernaut, but overly broad or hasty deployment could compound the harms of HR 1.

Addressing Medical Affordability Without Compromising Care
Kevin Schulman, Ezekiel J. Emanuel, MD, PhD
Quick ReadJul 09, 2026
Research
Your browser does not support the video tag.

The US spends 25 percent of all health care costs on administration, a big part of which involves billing for medical services. To realize savings requires standardizing and digitizing billing.

Research
Your browser does not support the video tag.

Addressing Medical Affordability Without Compromising Care

Kevin Schulman, Ezekiel J. Emanuel, MD, PhD
HealthcareRegulation, Policy, GovernanceQuick ReadJul 09

The US spends 25 percent of all health care costs on administration, a big part of which involves billing for medical services. To realize savings requires standardizing and digitizing billing.

Utah's Experiment With AI-Driven Prescription Renewals
Michelle Mello
Quick ReadMar 19, 2026
Research
Your browser does not support the video tag.

In January 2026, Utah announced a first-of-its kind pilot program allowing an autonomous artificial intelligence (AI) agent to renew prescriptions for consumers who request it. The state agreed not to enforce its unprofessional conduct laws against the developer, Doctronic, if the company adheres to a contract that includes safety and privacy protections. The pilot program includes 192 drugs for chronic conditions. Although physicians will initially validate the AI’s actions, the pilot program will swiftly become one of the first deployments at scale of an autonomous, agentic system in medicine. The announcement prompted concern from associations of physicians and pharmacists who opined that AI “should NOT be making care decisions.”

Research
Your browser does not support the video tag.

Utah's Experiment With AI-Driven Prescription Renewals

Michelle Mello
HealthcareRegulation, Policy, GovernanceQuick ReadMar 19

In January 2026, Utah announced a first-of-its kind pilot program allowing an autonomous artificial intelligence (AI) agent to renew prescriptions for consumers who request it. The state agreed not to enforce its unprofessional conduct laws against the developer, Doctronic, if the company adheres to a contract that includes safety and privacy protections. The pilot program includes 192 drugs for chronic conditions. Although physicians will initially validate the AI’s actions, the pilot program will swiftly become one of the first deployments at scale of an autonomous, agentic system in medicine. The announcement prompted concern from associations of physicians and pharmacists who opined that AI “should NOT be making care decisions.”