AI-Enabled Video System Monitors ICU Patients for Early Signs of Delirium

Computer vision experts have developed a 24/7 video monitoring system that tracks patient activity and helps doctors identify subtle signs of delirium.
Patients in intensive care units (ICUs) are surrounded by life-saving machinery, yet many experience a frequently overlooked state of confusion doctors call delirium. It’s among the most common complications for ICU patients, especially those on ventilators, and, left unchecked, it is linked to cognitive impairments and an increased risk of death. Yet because doctors on the front lines typically check patients only a few times a day, early signs of delirium go undetected. Ironically, the most severe form of the condition is actually a lack of activity, making it nearly impossible to spot using traditional question-and-answer methods.
Against that backdrop, a team of computer vision specialists and physicians set out to develop an AI-enabled ICU video monitoring system. Known as TRACER (Trajectory Reasoning Across Clinical Event Records), the system maintains a 24-hour vigil in the ICU. By integrating a patient’s electronic health records with continuous visual analysis, TRACER tracks their movements and evaluates how their activity changes over time. The team’s research was funded in part by the Stanford Institute for Human-Centered AI (HAI).
“TRACER helps doctors and nurses to better understand and ask questions about how patient behavior has changed over the course of the hospital stay,” said co-first author Zane Durante, a doctoral student advised by Fei-Fei Li, professor of computer science and a computer vision expert, and Ehsan Adeli, assistant professor of psychiatry and behavioral sciences. “Integrated with knowledge from the patient’s medical records, TRACER can help understand a patient’s physical state and map it to their mental states and human behavior over time to flag the possibility of delirium faster and more decisively.”
“This was a problem-driven solution, not a hammer in search of a nail,” said co-first author Paul Tang, a physician and adjunct professor at Stanford’s Clinical Excellence Research Center. “Most patients on a ventilator develop delirium during their ICU stay, and for many, cognitive impairment may last for years. By detecting signs of delirium early, TRACER could enable interventions that make a life-changing difference for patients.”
24/7 Observation
TRACER includes a queryable bedside video viewer designed to assist clinicians during rounds. The interface allows medical teams to ask questions, watch patient activities on a timeline, review drug and treatment data from the patient’s medical records, review footage that TRACER has flagged for human evaluation, and quickly navigate to timestamped video clips.
A new AI system called TRACER watches and analyzes days of ICU bedside video to help clinicians answer detailed questions about a patient's care — tracking movements, medications, and behaviors hour by hour to build a fuller picture of what happened and when.
“Our lab has long been interested in teaching machines to understand human behavior in long-form videos to help patients and clinicians,” said senior author Li. “Bringing that capability into the ICU is exactly the kind of high-impact setting where AI and computer vision can benefit humanity.”
The primary challenge to detecting delirium is the constraint on time and attention. During routine rounds – which typically last only a few minutes per patient – clinicians evaluate cognitive status by asking basic questions, such as What day is it? and Who’s the president? It can be difficult to detect delirium even when the patient is able to respond, but it becomes nearly impossible when they are incapacitated. Still, the TRACER team says vital indicators of a patient’s mental state are hidden in their movements – and, notably, in their inactivity – and critical to the system’s analytical approach.
Round-the-clock, AI-enabled video monitors that can “see” and interpret human motion while synthesizing written and numerical medical records offer a compelling alternative. The challenge, however, is wrestling with the sheer volume of data – both in terms of storage and the compute needed to analyze it. A TRACER system includes two cameras, trained on an ICU bed, that generate 15 frames per second of high-resolution video, 24 hours a day. Given that the average ICU stay lasts three-and-a-half days, and two cameras are needed to prevent the medical staff or patient from blocking the view, the resulting data volume is substantial.
The researchers mitigated these demands by reducing the amount of footage for processing and the sampling frequency. TRACER reviews a smaller subset of 15 “atomic actions” – defined as the smallest meaningful behavioral observations the system can make, such as hand-to-face movements, talking, or lying still – sampled at one-minute intervals. These observations are then compressed into hourly reports and visualized across the entire hospital stay, supported by the AI agent’s analysis of the patient’s health records. At the bedside, the doctor can query the system and review videos to assess patient activity and make a diagnosis.
“TRACER continuously analyzes video and clinical records to understand patient behavior over multiple days. Clinicians can then query patient activity through TRACER’s multimodal interface to support decision-making,” said co-first author Keshigeyan Chandrasegaran, a doctoral student advised by Fei-Fei Li and Juan Carlos Niebles.
TRACER includes reporting, timeline, video, and EHR tools to aid the medical team. In the paper, the authors provide an example of the voice-activated query tool in which the physician asks TRACER: “How did the patient’s behavior change across the past three days?” To which TRACER responds: “The patient’s activity peaked last night (8 p.m.–4 a.m.) with more sporadic activity earlier.”
Promising Prototype
In a test environment, TRACER achieved 85% action-detection accuracy and answered physicians’ queries correctly 61% of the time. An impressive result, the team acknowledges, that leaves them with room to optimize a promising prototype. Notably, in a cohort of 280 ICU patients, TRACER showed that those who screened positive for delirium exhibited markedly lower levels of “purposeful activity,” such as looking at others or communicating. That insight can be leveraged to refine the system in future iterations.
Importantly, the researchers stress that TRACER is “training free” – a plug-and-play model that understands human activity across patient populations at any ICU. It can also be easily updated as new models evolve. No patient data was used to train TRACER, allowing it to be shared with other ICUs without raising patient privacy concerns.
“This is an important distinction,” says physician Tang. “Building TRACER on a foundation model is intended to reduce the risk of one-off solutions that do not generalize well to other institutions – a persistent challenge for clinical AI.”
The research team is now refining the platform, and TRACER points toward a future where computer vision and AI improve clinical capabilities and extend observation around the clock, transforming how critically ill patients are evaluated, diagnosed, and cared for.
Read the preprint study, “TRACER: A Multimodal Harness for Patient Care in Hospitals.”





