2020 Seed Grant Recipients

Virtual Reality technology has shown great promise in visually immersing users in a variety of environments. However, these environments lack the rich auditory and haptic signals that humans expect during real-life interactions. Our long-term research goal is to create virtual experiences that provide rich multi-sensory feedback with applications in immersive journalism, storytelling, entertainment, education or e-commerce.  We will focus on the challenge of modelling how objects sound when users interact with them. While analytical models of sound exist, they assume precise knowledge of object material and shape, the environment, and user interaction. This makes it challenging to scale up an immersive virtual environment. Additionally, analytical models may be too computationally expensive for real-time feedback. These two challenges can be addressed through learning-based approaches. We propose to develop a generative audio model that is learned from a curated video dataset showing a person interacting with objects of various textures, materials, and physical properties. This model will generate realistic sounds for objects as users interact with them in a virtual environment. This approach promises to alleviate modelling challenges by leveraging data and generalising to new objects and user interactions that are not contained in the training dataset.

Our ability to interpret and explain algorithmic decisions is at the center of implementing AI safely and responsibly. In consumer lending, AI comes with the promise of expanding credit access to underserved populations. But machine decisions have to be transparent to detect and mitigate discrimination, assess model robustness across the business cycle, and provide legally mandated explanations to consumers applying for credit. To deploy algorithms in a sensitive area like consumer lending, we postulate that model explanations must diagnose the specific societal impact and economic consequences an AI model has. In this project, we first develop a theoretical framework that derives desired properties of explainability techniques given regulatory objectives in consumer lending. We then propose a step-by-step approach to empirically evaluate the ability of explainability techniques to assess model behavior with respect to those regulatory objectives. Third, we demonstrate the utility of our proposed framework by applying it in a study involving commercially available explainability tools that use a variety of techniques for supplying information about a model’s functioning. Through our partnership with FinRegLab, an independent, nonprofit research organization with extensive expertise in financial policy, regulation, and compliance, we ensure the inclusion of a diverse and interdisciplinary group of stakeholders in formulating research questions, incorporating current legal and regulatory conditions, conducting the evaluation, and disseminating our research findings. This effort is designed to inform and complement ongoing efforts to develop standards for the safe, fair, and responsible use of ML/AI.

As robotic hardware and intelligence improve, robots are becoming more effective teammates with humans. One collaborative task is cooperatively carrying a heavy object. When multiple humans carry a large or heavy object together, they reduce the risk of individual injury and leverage cues from other teammates and the environment to work cooperatively. This cooperation is usually characterized by efficiency: moving quickly toward a goal and not using more force than necessary. The goal of this work is to make a robot an effective teammate in this task, where at least one of the other teammates are human. By having a robotic teammate, the same advantage of mitigating personal injury remains. The challenge is teaching the robot how to leverage cues from the human teammate(s) and the environment in order to carry efficiently. To accomplish this, we plan to leverage behavioral cloning with emerging techniques in machine learning to teach the robot how to be an effective teammate in this cooperative transport task.

We are increasingly surrounded by autonomous mobile robots that augment our capabilities in a variety of applications including household cleaning, package delivery, and security monitoring. These robots operate on a variety of uncertain information, including noisy sensor data and probabilistic decision making processes. In their current state, they often fail and require users with limited training to intervene and troubleshoot the problem. To help users effectively troubleshoot robots, it is important to present relevant information and its uncertainty in an interpretable form without causing information overload. We study how humans reason about spatiotemporal data under uncertainty and subsequently strategize using the information available to them. We then leverage this understanding and explore the design of adaptive interfaces that can assist users in better troubleshooting robots.

Significant lifetime benefits in human health are conferred from exposure to responsive parenting behaviors during early childhood. After adjusting for sociodemographic factors, such early exposure is associated with positive socioemotional, cognitive, and physical health outcomes through adulthood. As a result, much early child-development research requires carefully curated video-taped observations of parent-child interaction. These studies, however, typically employ human annotators to code these video observations manually, and because the process is so time-consuming and labor intensive, study findings have been limited by small and less diverse samples. In this project, we propose to develop an AI-based tool for automated coding of video capture of parent-child interaction, in order to advance measurement detail and scale in child-development research. Once we develop the AI-based tool, we will assess its validity by conducting a longitudinal study of patterns associated with responsive parent-child interactions, using the Study of Environmental Effects on Developing LINGuistic Skills (SEEDLingS) dataset, which captures video encounters of parent-child interactions over time.

What do political micro-targeting, platform content moderation, and the use of stingray and facial recognition technologies by local police forces have in common? Each of these AI-enabled practices reveals the role of third-parties in people’s ability to come together to take action. The Digital Civil Society Lab (DCSL) will develop a path-breaking, multi-disciplinary and collaborative volume exploring how artificial intelligence impacts a cornerstone of democratic life: the people’s ability to assemble. The manuscript (tentatively titled Artificial Intelligence, Real Democracy: How AI changes people’s ability to assemble, and what it means for democracy) will be written through a workshop process designed to produce a volume organized around a few central questions and that stands as an example of collaborative research production. The research team will also draw on our Digital Assembly Research Network (DARN) - a global community of 350+ scholars, technologists, policy makers and civil society actors dedicated to addressing the ways digital systems influence our ability and right to assemble. The volume and the DARN together serve to integrate policy, practice, and scholarship.

Each day, cable TV news networks determine what information millions of Americans receive. They also set the context and tone in which that information is presented. These decisions shape public opinion and culture. In this project we aim to develop computational tools that provide greater transparency about these editorial choices. Specifically, we will use modern deep-learning based image, text, and audio processing techniques on a decade of nearly 24-7 broadcasts from CNN, MSNBC, and Fox News to identify patterns and trends in content, bias, polarization, and editorial choices.  We will also develop new systems for quickly validating the accuracy of these machine annotations.

We will work closely with media watchdogs, journalists and other researchers to perform in-depth, data-driven analyses of specific areas of bias in cable TV news coverage. These efforts will address questions like: How much screen time is given to specific topics, and how does this differ across shows or channels? What is the gender and race distribution of expert commentators asked to speak about key topics? Our goal is to discover where such biases appear and to show the potential of AI to aid large-scale quantitative analysis of the news.

This project focuses on creating transformative algorithms, solutions, and tools for human-AI augmented control of facilities. It will fulfill a critical need for various next-generation establishments, ranging from power grids to industrial-scale factories to nuclear plants. As these grow in complexity, their control and operation exceed human operators' abilities.

For such applications, human-AI augmentation is essential. Still, pure AI approaches create additional risks, with black-box AI solutions having uncertain validity and robustness. Our research paradigm focuses on using AI-agents not to supplant but to supplement human operators. However, the introduction of a highly automated AI-agent can cause "out-of-the-loop'' syndrome, whereby operators become increasingly detached from the system and may lose their ability to act when the AI fails or becomes unavailable. Also, a critical weakness of AI prediction systems is their opacity and lack of interpretability. Consequently, our focus will be on designing an AI system that augments human operators but leaves them in charge through interpretability, transparency, and explainability.

Recent developments in AI—including the creation of OpenAI’s GPT-3 autoregressive language model—have improved language-generating capabilities in tasks such as translation, question-answering, and content creation. Earlier generations of OpenAI’s technology, such as GPT2, have already been used to write emails to colleagues and blog posts by venture capitalists, and similar technologies may be used by malicious actors for disinformation campaigns. The extent to which GPT-3 poses a threat to U.S. public discourse depends partly on persuasiveness and detectability. In this study, we will conduct multiple survey experiments comparing the persuasiveness and perceived credibility of GPT-3 created content, content created by known disinformation actors, and content created by people sharing their authentic beliefs. We plan to vary topics, content length, and content format in order to assess the conditions under which AI-created text is likely to be effective. We will also look at the effect of learning about the abilities of GPT-3 on general trust in media. If GPT-3 can produce content that is nearly as, or more convincing than, human-created content, GPT-3 may pose a significant risk to public discourse and the quality of the information environment.

According to the U.S government, the persistent spread of fake news and other types of misinformation is one of the main on-going threats to societal cohesion and trust. This misinformation is often disseminated by computational “bots” that first learn about specific social media landscapes and then tailor fake news to be “emotionally and culturally resonant” with those landscapes. Thus, while there is a clear need for digital defense tools, their effectiveness hinges on a deep understanding of why certain affective content garners increased engagement or is “culturally resonant”. Unfortunately, we know relatively little about how affective and cultural factors shape the spread of misinformation on social media. Based on our previous research (Hsu et al., 2020), we propose a values-violation account of virality, in which we predict that the affective content that is most likely to spread is that which violates dominant cultural values regarding emotion. According to this account, high arousal negative content spreads the most in the U.S. because it violates the American value placed on maximizing positive feelings and minimizing negative ones. To further test this account, we propose to: (1) build natural language processing tools to examine and compare the spread of affective content on social media, and (2) develop algorithms capable of supporting “affective filters” that can be deployed on social media platforms to flag or modify affective content, providing a culturally-adaptable defense against affectively viral misinformation. We will also examine affective virality in specific subgroups within each country. These studies will not only advance AI research by integrating machine learning algorithms with human emotion and culture, but also advance our understanding of the kinds of affective propaganda users are most vulnerable to so that organizations individuals can defend against them.

Readmission of patients after discharge from hospital is common and often costly. In the United States, one-fifth of the discharged patients (nearly 2.6 million seniors), face an acute medical complication within 30 days that requires readmission or emergency department (ED) visits. In this project, we propose strategies to reduce this readmission rate, especially among older adults. We introduce an intervention approach based on the monitoring of Activities of Daily Living (ADL) trajectories using the computer vision (CV) technology.

Over the last several years, researchers and practitioners have worked to design fair prediction models, used in criminal justice, health care, social services, and beyond. Less attention, though, has been paid to the inextricable link between predictions and decisions. In this project, we aim to develop methods for creating equitable prediction-decision systems that respect the often complex preferences and constraints that arise in many applied problems. We will develop our conceptual framework within the context of a high-stakes application: designing online advertising strategies to recruit participants for CalFresh, California's Supplemental Nutrition Assistance Program (SNAP), which helps people with lower incomes buy nutritious food. Through scientific advancements in fair machine learning, we have an opportunity to substantially improve food security.

With the COVID 19 pandemic, the live performance industry has been shut down.  Many audiences and performers find that Zoom performances are unsatisfying both due to the constraints of web cameras and to the slow network speed. However, there is an unexplored area of technology that could be applied to improving streamable live performances, one that utilizes machine learning algorithms to generate “virtual performers” that are controlled by a live performer. We approach working with these algorithms from a different perspective—to think about these systems of technology from a poetic and creative standpoint. Can we improve theatrical streamed performances using machine learning? What are the creative affordances that appear when we apply artificial intelligence to the representations of characters within a live performance? Our project will undertake research both in a creative field by exploring the aesthetics of a “digital performer” alongside a rigorous scientific analysis of this networking protocol and technology behind it.

Wildfires are increasing in prevalence and severity, causing incredible human suffering and damage to property. In 2009-2018, 7 million acres burned in California, double the amount from 1979-1988. To combat wildfires in the Western US, ALERTWildfire has deployed 600 remotely operable pan-tilt-zoom cameras that have the potential to enable early detection of wildfires. Unfortunately, human monitoring is too expensive for these vast networks of cameras. We will design statistical algorithms and a runtime system to aid human operators in monitoring these camera networks for early wildfire detection. Designing such algorithms and systems must address key challenges: the need for low-latency, early detection, limited resources in the form of computational resources and human attention, and constantly changing environments in nature. We hope to allow scaling to tens of thousands of cameras, enabling early responses to potentially devastating wildfires.

The role of AI in providing us content online through social media feeds and search engine results is critically important—from consumer choices to political opinions, we know that these algorithms can change our beliefs and behaviors. But they often do so in unpredictable and invisible ways; studying AI-mediated content is a complex challenge, since most of these algorithms are proprietary and inaccessible to researchers and the public, and users might interact with dozens of different systems online every day. Our team will spend the coming year building a system to help users understand the role of AI in the online content they consume, and measure the impact this content has on them, specifically in the domain of news media and political information. To do this, we are building an in-browser tool that combines a web application and browser extension to measure the algorithmic content users receive, make insights about this content visible to them, and deploy interventions and surveys to understand how this affects them. This research will highlight the way AI systems influence us through our media environments, and the impact of making that influence more transparent, with the goal of promoting the development of AI-powered online systems that are interpretable and human-centered. 

Access to quantitative, robust, yet affordable diagnostic tools is necessary to reduce global infectious disease burden. Manual microscopy has served as a bedrock for diagnostics with wide adaptability, although at a cost of tedious labor and human errors. Automated robotic microscopes are poised to enable a new era of smart field microscopy but current platforms remain cost prohibitive and largely inflexible, especially for resource poor and field settings. Here we present Octopi, a low-cost and reconfigurable autonomous microscopy platform capable of automated slide scanning and correlated bright-field and fluorescence imaging. With roughly two orders of magnitude in cost reduction, we aim to develop an Octopi based large robotic microscope network for improved disease diagnosis while providing an avenue for collective efforts for development of modular instruments and machine learning algorithms for better disease diagnostics for the masses.

Deep neural networks have emerged as leading models for predicting neural data from a variety of brain areas and species. We will explore whether this modeling framework can be used to predict how neural representations of visual stimuli change over development. Like the human visual system, deep network models of the adult visual system are created from a combination of pre-specified “hardware” and an extensive period of exposure to visual images.  We will thus approach the modeling in two ways, one is to build models that represent the initial conditions of the visual cortex prior to the onset of visual experience and the second is to use the training phase of models that differ in their architectures and training rules as models of human brain development.  Critically, to test these models, we will acquire rich, high temporal resolution data sets from developing human infants using high-density EEG recordings.

Illegal fishing is a pressing global problem that threatens food security, livelihoods, ecosystems, and robs governments and communities of billions of dollars in income. Current approaches focus on detecting illegal fishing by commercial vessels in international waters. We propose to develop an AI-based algorithm to detect fishing vessels of all sizes, using computer vision to automatically analyze satellite imagery, focusing on nearshore areas and ports that are not well-studied. By pairing these data with available GPS-based data from vessel tracking systems, we aim to paint a comprehensive picture of vessel activities in places where fisheries are central to local communities economies and well being. With this research, we aim to address key gaps in currently available data and understanding, and highlight priorities for future studies to improve fisheries transparency and eliminate illegal fishing.

Physicians face an exponential influx of medical data. The lack of computerized assistance places an immense cognitive burden on physicians who must manually assimilate, review and interpret these data to make clinical decisions leading to treatment variability and physician burnout. Artificial intelligence (AI) techniques show promise to alleviate these problems but suffer from two main limitations: (1) They require enormous quantities of labelled data for training. Curating large labeled datasets is time-consuming and expensive; and (2) training labels are often simplistic and task-oriented (e.g. normal vs. abnormal). Even state-of-the-art AI models trained on such labelled data often fail to generalize to unseen real-world patients. The goal of this project is to develop new methodology for transferring acquired human insights and knowledge to machine learning models in the context of their providing healthcare, stepping away from classic supervision using simplistic labeling or specifying explicit rules, by simply observing skilled humans in action. We introduce a novel paradigm for developing more robust AI models that we call “observational supervision,” wherein we train AI models by observing how experienced physicians examine medical data while making decisions. Our approach not only requires minimal, if any hand labelling of data, but the signals derived from physicians’ actions (e.g. what data they look at, the order in which they look, length of time, etc.) also reflect the physician’s underlying thought process that can be injected into model training. Moreover, our work will be applicable to and benefit many other domains beyond medical care, such as augmented reality - virtual reality (AR-VR), sensor fusion, environmental monitoring, among others. 

Design sketching is both a fundamental skill in artistic expression and a powerful form of visual communication in its own right. However, outside of classroom environments, tools supporting design sketching cannot yet provide feedback and critique necessary for learning. We propose a joint human-AI approach to providing student feedback on design sketching, thereby augmenting human capacities to learn drawing and design sketching. We will develop Korekta, a human+AI corrective feedback system that balances evaluation between AI---for the skills that can be classified algorithmically---and humans---for the skills that cannot. We will further demonstrate how Korekta's correction process generates training data that helps its AI improve itself as a side effect.

In the wake of George Floyd’s death, calls for companies to fight racism and racial bias in their organizations and products have escalated. In the tech world, machine learning algorithms that reproduce, reinforce, and even amplify racial disparities and biases in society have been identified as key sites for intervention. Our team of social psychologists, computational linguists, and computer scientists will collaborate to develop a series of AI tools that will detect and root out bias rather than magnify it.  Working with Nextdoor, a neighborhood-focused social network, we plan to pair our bias detectors with behavioral nudges and interventions that encourage inclusive rather than exclusive behavior among users on the platform. With these detectors, we will be able to experimentally test and iterate multiple interventions across a large number of users, leveraging the strengths of AI and social science to support positive behavioral change and strengthen communities.

One’s voice is arguably one of the most human aspects of a person. Unfortunately, patients with laryngeal cancer often undergo laryngectomies (or surgical removal of the voice organ) and consequently suffer permanent loss of their voice. Current methods for voice rehabilitation rely on modulation of artificially produced vibrations, which results in a voice that is unnatural, monotonous and difficult for an untrained listener to understand. We aim to develop a process to naturalize and individualize the post-laryngectomy voice. More specifically, our approach will be to apply machine learning methods to perform audio-to-audio translation of post-laryngectomy speech, utilizing the unique characteristics of the patient’s pre-laryngectomy voice. We anticipate that any meaningful voice naturalization will help patients regain their individuality, enhance communication, and improve overall quality of life.

Opening the Loop is the tentative name for a documentary film being directed and produced by Muhammad Khattak (Stanford) and Joe Khoury (California Institute of the Arts). The film seeks to critically interrogate how broader issues regarding power, culture and our ethical priorities influence current developments of artificial intelligence. Primarily focusing on AI’s regulatory uses, it aims to shift away from predominant attitudes that technology is merely a neutral tool and focuses on AI’s slow expansion into the public realm. This is part of a broader project to promote non-technical engagements in AI. In conjunction with the film, we will be collaborating with Stanford artists and educators to organize an AI art exhibit and high school outreach program about the societal implications of AI. The latter is meant to introduce ethical questions about AI to the realm of secondary education whereas the former serves to promote unique engagements in current technological changes. Together, these three projects focus on the particular experiences that are oftentimes marginalized in popular representations of AI and aim to endorse more open-ended, philosophical and artistic means of describing AI in current conversations.

Rapidly developing technologies that use artificial intelligence can be an unprecedented force for good; however, they can also codify and amplify existing forms of racial inequality, discrimination, and bias. The Technology and Racial Equity Action Lab, based at the Center for Comparative Studies in Race and Ethnicity, combines research, teaching, and practice to advance racial justice in the analysis, production, and deployment of new technologies. The Lab will produce public-facing reports, policy briefs, and recommendations on critical issues at the intersection of race and technology. In addition, the Lab expands these impacts through its international Practitioner Fellows Program that brings together these external fellows with Stanford students and faculty.

More than sixty years after “Brown v. Board of Education”, segregation by race and class is still a growing problem in public schools in the US. Many school districts adopted student assignment mechanisms in order to provide families more options while hoping also to disentangle neighborhood segregation and school segregation. Although families express their preferences in such mechanisms, priorities and zones can be designed by the district to achieve societal objectives. Ad-hoc design choices have been incorporated by districts to address these issues, such as assigning priority to students from neighborhoods with low income or a high percentage of minorities. However, there is little evidence that such policies in the context of choice have helped to address concerns about equity. In 2018, the San Francisco Unified School District passed a resolution for developing a new student assignment system for elementary schools, which seeks to improve diversity, transparency, and equal access to quality schools. This project will build tools from AI, optimization and Economics to design mechanisms, priorities and zones towards achieving these goals.

Explanations are critical to how humans understand and learn about the world. As humans, we readily go beyond what happened to reason about why it happened and how things could have played out differently. While today’s AI systems can achieve super-human performance on many challenging tasks, their lack of generalizability, interpretability, and ability to interact with humans limits their potential, especially in high-stakes settings. One key aspect of human intelligence that is missing is the ability to understand and communicate about causality. Drawing inspiration from how humans think and communicate about causality, the Science and Engineering of Explanations (SEE) project pursues the twin goal of developing AI systems that generate and understand causal explanations the way humans do, and that help to improve human explanatory abilities.

Take a moment to attend to the varying degrees of control you have over your surroundings. Your limbs are decidedly part of you: you can control them directly, sensing them through high-bandwidth proprioception. Further along this "agency spectrum" are tool-like objects such as forks, which you can grab, control, and use as extensions of yourself, albeit without proprioceptive feedback. Other objects, like a table, can be manipulated, yet the act of doing so is less direct than with tools. Then there are other agents, which you can affect even less directly, such as through social interaction. Finally, there are aspects of the environment that affect you but cannot easily be affected by you, such as the weather, and yet other aspects which neither affect you nor can be affected by you, such as far-off buildings or background noise. 

We hypothesize that two major mysteries regarding agency are strongly connected. The first is computational: how do we design an artificial system with a complex, capable embodiment that learns how to efficiently leverage the full spectrum of agency available to it in a natural manner---that is, through exploration, and without explicit supervision? The second problem is cognitive: humans represent an agentic self as a distinct entity from the world. What is the utility of this self: why and how do we cognitively track it in this distinguished way? We strive to answer these computational and cognitive questions, while also elucidating the underpinnings of dissociation-- a psychiatric disruption of self implicated in prevalent stress and trauma-related mental disorders.

Progress on the UN Sustainable Development Goals (SDGs) is hampered by a persistent lack of data regarding key social, environmental, and economic indicators, particularly in developing countries. While there have been numerous attempts to fill this gap through the combination of machine learning and satellite imagery, a key challenge in many sustainability applications is the lack of sufficient labeled training data. In particular, geospatial analysis lacks methods like the word-vector representations and pre-trained networks that significantly boost performance across a wide range of natural language and computer vision tasks. To fill this gap, we propose to extend recent advances in contrastive self-supervised learning to remote sensing data. On natural images, these techniques can learn in an unsupervised way (i.e., without human annotations) representations that perform close to their supervised counterparts. By extending these techniques to satellite imagery, we expect to be able to learn representations that can significantly improve performance in downstream classification tasks such as poverty and food security estimation.

Much of our visual world is highly regular: objects are often symmetric and have repetitive parts; scenes may consist of objects organized in a repetitive layout. The goal of this project is to build machines that infer and represent such regular structures from raw visual data and later leverage them for scene understanding and analysis. The key technical innovation will be methods that integrate program representations into modern deep networks.

Stanford University´s Human Trafficking Data Lab is a multi-stakeholder effort to develop scalable technologies that leverage the modern data economy to find human trafficking and support interventions for those who have been enslaved. Our Lab is assembling powerful anti-trafficking data resources to create a unique decision support system combining near real-time machine learning insights with AI technology capable of proactively identifying forced labor and trafficking. This project will further enable us to crosslink satellite imagery with cases of illegal labor camps, producing open-source tools and a newly digitized inspection report cache, in order to disrupt systems of exploitation. 

Sea level rise and its associated effects will be among the most immediate and dramatic impacts of climate change in this century. Small changes in mean sea level increase the rate of high sea level events that can have devastating impacts on coastal communities. Despite its importance, sea level rise projections have significant error margins, in large part due to high uncertainty in the response of Greenland and Antarctica to a changing climate. This is a result of sparse measurements, high measurement uncertainty, and incomplete understandings of the governing physics of ice sheets. Advances in autonomous aircraft promise to enable continuous data collection campaigns over ice sheets previously considered logistically impossible, but even a huge volume of data will not resolve current uncertainties if it is collected without regard for the uncertainty in the governing physics. Building on techniques from scientific machine learning and Bayesian inference, we are investigating an uncertainty-aware approach to selecting measurement locations to maximize the value of autonomous data collection aircraft in improving future sea level rise estimates.

During the COVID-19 pandemic, health care resources are stretched to the limit. Lack of testing capabilities and proliferation of “worried well” patients have put pressure on clinicians to prioritize needs at healthcare facilities. An efficient and effective way to triage patients has become increasingly crucial.

We aim to develop and validate a novel video-based AI algorithm to automatically triage patients with COVID-19 concerns based on a combination of their clinical appearances and their reported symptoms during a telehealth video encounter. The novel algorithm can help determine which patients need emergency assistance, which patients need to come to the clinic for further evaluation, and which patients are safe to stay home.

If successful, the novel algorithm can enable large-scale triage of patients with COVID-19 concerns without using clinicians’ time, thus using digital technologies to boost the health system's capacity. In addition, this video-based self-triage tool could potentially reduce healthcare disparity in underserved populations given their poor access to in-person providers but relatively high rate of mobile phone and internet usage. Finally, the video-based algorithm can be broadly applied to other respiratory outbreaks and diseases beyond COVID-19.