AI Audit Challenge 2023 Finalists
Most data collected from the real world is biased, unbalanced, and contains sensitive and private information. This reality has given rise to the idea of creating synthetic datasets to alleviate bias, harm, and privacy concerns inherent in the actual data. This concept relies on Generative AI models to produce unbiased, privacy-preserving synthetic data. How can we tell if this approach delivers on its promises? How can we determine whether a synthetic dataset and a model built from it conform to fairness, privacy, and fidelity requirements and is still valuable for the downstream task? We present a real time synthetic data auditing platform that offers a holistic assessment of synthetic datasets and AI models trained on them centered around bias and discrimination prevention, fidelity to the real data, utility and privacy preservation. Our platform connects different stakeholders (data scientists, data governance experts, internal reviewers, external certifiers and regulators) from model development to audit and certification via a synthetic data auditing report. We believe that such external transparent reporting should become an accepted process to facilitate bias and discrimination detection and to ensure compliance with policy requirements, safety and performance guarantees.
Name Affiliation Youssef Mroueh (Lead) IBM Research Pierre Dognin (Co-Lead) IBM Research Inkit Padhi (Co-Lead) IBM Research Brian Belgodere IBM Research Adam Ivankay IBM Research Igor Melnyk IBM Research Aleksandra Mojsilovic IBM Research Jiri Navartil IBM Research Apoorva Nitsure IBM Research Mattia Rigotti IBM Research Jerret Ross IBM Research Yair Schiff IBM Research Radhika Vedpathak IBM Research Richard A. Young IBM Research The tool that audits third party ML APIs was built and open-sourced by marrying two different ideas together: (1) system status pages, which Engineers use to monitor the uptime of APIs, and (2) model cards, that ML Researchers use to report on the performance of ML models. The result is an AI audit bot: it runs once a week and uses an open dataset to evaluate the performance of Google’s sentiment API and publishes its results online as a timeline of metrics. Extending this type of prototype to cover a wide range of proprietary AI APIs can give researchers and practitioners valuable data and alerts about how the models behind them are changing over time.
Name Affiliation Neal Lathia Monzo Bank, UK Meerkat is an open-source Python library that provides computational infrastructure for performing participatory algorithmic audits over unstructured data. It is designed to address two key challenges faced by auditors. (1) Auditors perform data wrangling operations to compute statistics on protected subgroups, but these operations are often difficult to express when working with the unstructured data types commonly found in ML (e.g. images, video, long-form text). (2) Auditors struggle to include input from end-users and domain experts who lack technical proficiency, because existing audit tools lack no-code interfaces.
First, Meerkat extends familiar data science abstractions (e.g. data frames) to unstructured data types (Challenge 1). Second, Meerkat provides a flexible framework for composing graphical user interfaces to interact with data (Challenge 2). Together, these enable participatory audits where technical auditors can manipulate data in code, while involving non-technical stakeholders through custom GUIs.
We demonstrate how Meerkat facilitates audits in three distinct settings: (1) auditing the popular DeepFace facial verification library to characterize performance degradation due to racial identity; (2) auditing the popular Stable Diffusion image generation model to understand its conception of gender and race; (3) auditing updates to three ML as a Service (MLaaS) Vision APIs by generating change lists that characterize their impact on downstream users. Interested in using Meerkat for your applications? Check out our website and join our Discord!
Name Affiliation Karan Goel (Co-Lead) Stanford University Sabri Eyuboglu (Co-Lead) Stanford University Arjun Desai (Co-Lead) Stanford University James Zou Stanford University Chris Ré Stanford University As computer vision technologies are increasingly being used in various social aspects, it is crucial to understand the limitations and biases of these systems. However, one of the fundamental challenges in auditing vision models is the scarcity of data for underrepresented demographics. For instance, when developing a visual classifier to detect lipstick-wearing individuals, the model might inadvertently acquire a pronounced gender bias, given the higher frequency of women wearing lipstick than men. Consequently, the model could erroneously rely on gender as a deceptive indicator, resulting in misclassifications for both lipstick-wearing men and lipstick-free women. Obtaining sufficient data to mitigate this issue, such as images featuring men wearing lipstick, is challenging.
In this work, we address this data conundrum by investigating the use of language to audit vision models. Language, with its inherent flexibility and compositional nature, enables the generation of a vast array of diverse and novel inputs. These generated language inputs help us understand when the model fails by discovering error slices (e.g., the model struggles with images of men wearing lipstick), demystify why the model fails by identifying the most influential attributes (e.g., the presence of a woman is strongly associated with lipstick), and eventually rectify these failures (e.g., the model should be able to recognize men wearing lipstick). We provide a Jupyter Notebook that demonstrates how our method effectively addresses the aforementioned scenario, showcasing the practical application of our approach in overcoming biases and limitations in computer vision models.
Name Affiliation Yuhui Zhang (Co-Lead) Stanford University Serena Yeung (Co-Lead) Stanford University Jeff Z. Hao Chen Stanford University Shih-Cheng Huang Stanford University Kuan-Chieh Wang Stanford University James Zou Stanford University Ceteris Paribus is a tool for on-the-fly, interactive, and personalized discrimination auditing of large language models. Our user-friendly solution allows users with no computational expertise to run audits that are fully customized to their needs and created from scratch.
An auditor simply has to indicate their topic of interest and a list of protected groups. Without any pre-existing data, our tool potentially finds countless examples of the language model being discriminatory to those protected groups on the topic of interest. The auditor is intermittently asked to guide the audit in the direction they would like. They can select examples they find interesting and cross out those that are off-topic, to push the tool to find more of the former and less of the latter.
Our hybrid approach allows us to merge the benefits of precise human judgment and large-scale automation. The auditor in the loop ensures that the audit is grounded in prompts that adhere to the standards of the organization requiring the audit, such as legal firms or policymakers, while automation ensures that endorsed examples can be rapidly expanded upon and that the landscape of discrimination is exhaustively explored, surfacing examples of discrimination that even the auditor would not have anticipated. Our ecosystem of widgets automates the manually-tedious extraction of coarse-grained discrimination trends from individual examples.
Overall, Ceteris Paribus empowers users to take a proactive approach to discovering discrimination in language models by providing a simple and effective solution that can be tailored to the needs of any individual or organization.
Name Affiliation Adarsh Jeewajee (Co-Lead) Stanford University Edward Chen (Co-Lead) Stanford University Xuechen Li University of Pennsylvania Ransalu Senanayake Stanford University Mert Yuksekgonul Stanford University Carlos Guestrin Stanford University There is an increasingly large ecosystem of AI evaluation tools, moving beyond performance to robustness, fairness, and other behaviors. This trend is welcomed by the responsible AI (RAI) community, but introduces new strategic and technical challenges. On the strategic side is the difficulty aligning on and operationalizing acceptable AI behavior across an ever-growing range of system characteristics. On the technical side, practitioners need to be aware of assessment tools and sufficiently skilled to use them.
To address these challenges, we developed an AI governance solution that provides “policy packs”: sets of standardized controls to be implemented by AI developers and MLOps tools. Our approach is founded on a RAI governance platform, a SaaS product which allows AI governance stakeholders to apply policy packs to AI systems and create reports that provide transparency into a system’s behavior and compliance with selected policies. A library of policy packs addressing common AI system use cases and governance contexts enables scalability.
Complementing the Platform is Credo AI Lens, an open-source assessment framework that addresses technical challenges by giving developers a single tool to access diverse AI assessments. Lens is directly integrated with the Platform, enabling a direct connection between governance requirements and technical evaluation.
The combination of the Platform and Lens has the potential to scale and become widely adopted in the AI governance ecosystem, creating a more standardized and transparent evaluation process for AI systems. We showcase this future by recreating the ProPublica audit of the COMPAS system using our system.
Name Affiliation Ian Eisenberg Credo AI Ehrik Aldana Credo AI CounterGen is a framework for auditing and reducing bias in natural language processing (NLP) models such as generative models (e.g. ChatGPT, GPT-J, GPT-3 etc.) or classification models (e.g. BERT). It does so by generating balanced datasets, evaluating the behavior of NLP models, and directly editing the internals of the model to reduce bias.
CounterGen lets you generate variations of your data by changing protected attributes, which should not affect the model output. This makes it possible to measure and mitigate the bias in the actual text your model processes in deployment, which is not possible with traditional benchmarks. It also allows tackling the root cause of bias, the data itself, rather than only patching it after the training, i.e. via fine-tuning or filtering.
CounterGen is easy to use, even by those who don’t know how to code, and is applicable to any text data, including niche use cases, and it proposes concrete solutions to debias biased models.
CounterGen's Python modules and online tool are free, open-source, and easily accessible for organizations aiming to provide their members with bias evaluation tools. Comprehensive documentation and initial findings from the use of these tools are available for reference.
Name Affiliation Fabien Roger (Lead) SaferAI Simeon Campos SaferAI Crowd-Audit enables experts to conduct quicker and more thorough audits by facilitating the crowdsourcing of deficiencies by end (lay) users.
This tool helps address two common issues with audits: that they 1) take a long time, and 2) are conducted by mainly experts. This long process comes from the fact that auditors have to individually test hypotheses before identifying areas for improvement. With this long process, many AI systems in need of auditing are not audited. In addition, given experts' limited time and small number, audits are limited by the cases the experts decide to test.
Crowd-Audit addresses both problems by introducing a tool that will make audits faster to conduct and more thorough. It features a web-application that collects multiple end (lay) users’ audits of a system. Crowd-Audit also has an aggregator that condenses the multitude of user reports into a single report that enables experts to see areas of improvement.
While both the web-application and aggregator are currently focused on Perspective API, it can be modified to extend to other open source AI applications. (Perspective API estimates a comment’s toxicity and is used in content moderation on social media, such as in the New York Times.) This extension is particularly easy for other NLP-based algorithms.
Name Affiliation Christina Pan Stanford University Audits have risen as powerful tools to hold algorithmic systems accountable. But because AI audits are conducted by technical experts, audits are necessarily limited to the hypotheses that experts think to test. End users hold the promise to expand this purview, as they inhabit spaces and witness algorithmic impacts that auditors do not. In pursuit of this goal, we propose end-user audits—system-scale audits led by non-technical users—and present an approach that supports end users in hypothesis generation, evidence identification, and results communication. Today, performing a system-scale audit requires substantial user effort to label thousands of system outputs, so we introduce a collaborative filtering technique that leverages the algorithmic system's own disaggregated training data to project from a small number of end-user labels onto the full test set. Our end-user auditing tool, IndieLabel, employs these predicted labels so that users can rapidly explore where their opinions diverge from the algorithmic system's outputs. By highlighting topic areas where the system is underperforming for the user and surfacing sets of likely error cases, the tool guides the user in authoring an audit report. In an evaluation of end-user audits on a popular comment toxicity model with non-technical users, participants both replicated issues that formal audits had previously identified and also raised previously underreported issues such as under-flagging on veiled forms of hate that perpetuate stigma and over-flagging of slurs that have been reclaimed by marginalized communities.
Name Affiliation Michelle S. Lam (Lead) Stanford University Mitchell L. Gordon Stanford University Danaë Metaxa University of Pennsylvania Jeffrey T. Hancock Stanford University James A. Landay Stanford University Michael S. Bernstein Stanford University The existing issue of bias in text-to-image generators call for continued effort towards auditing and debiasing models. Our auditing tool, FACIA (Facial Adjectival Color and Income Auditor), contributes to transparency and awareness for both end users and developers by automating the inspection of representational bias in the generation of images. FACIA specifically uncovers occupation, skin lightness, gender, and trait sentiment bias apparent in results when the model is provided with human-centric but ambiguous and non-gendered language.
FACIA’s workflow is encapsulated within its three subcommands: “generate”, “analysis” and “evaluate”.
Generate automatically generates trait and occupation prompts that evenly sample across sentiment categories for trait descriptive adjectives and median income spaces for occupational titles.
Analyze takes the images generated by the prompts and applies image equalization, face detection, gender prediction, skin color extraction, and skin lightness transformation. Evaluate compares the skin lightness and gender tendencies against trait sentiment and income distributions to statistically assess whether a model generated biased images with respect to skin lightness and gender.
FACIA is publicly available on GitHub to be run as a command line tool and Python package. Alternatively, a UI demonstration of the tool and past audit results are available on Hugging Face.
Name Affiliation Valeria Rozenbaum (Lead) Thomson Reuters Special Services (TRSS) Zach Seid Thomson Reuters Special Services (TRSS) Etienne Deprit Thomson Reuters Special Services (TRSS) Vivian Tran Thomson Reuters Special Services (TRSS) Spencer Torene Thomson Reuters Special Services (TRSS) Xena Grant Thomson Reuters Special Services (TRSS) Kyle Soto Thomson Reuters Special Services (TRSS) AI models have become well integrated into our daily decisions, yet a simple and clear way to detect and prevent discrimination is still missing. Policymakers and data consumers are constantly faced with evaluating biases caused by technical AI models that even experienced data scientists struggle to quantify.
Framework for Responsible AI Model Evaluation (FRAME) seeks to provide a framework for testing biases in open-source models with scalability and replicability in mind. Coupled with a curated dataset with focus on various bias categories including gender, race, religion, and profession, we demonstrate the effectiveness of FRAME on black-box models similar to the brains behind ChatGPT. Our demo has a use case in resume screening because that is the most impactful and most immediately applicable potential use case for adopting our framework.
Today, non-technical users of AI and policymakers are at risk of issues from unintentional discrimination that arise in various stages of the decision making process. FRAME offers a solution that makes it easier for anyone to test and visualize potential biases by addressing technical challenges around data, models, and metrics. Ultimately, FRAME seeks to provide a useful framework for all the stakeholders in the AI bias audit process.
Name Affiliation Kay Lim University of Chicago Booth School of Business Keon Kim University of Chicago Booth School of Business Hate speech detection models play an important role in protecting people online. When they work well, they keep online communities safe by making content moderation more effective and efficient. When they fail, however, they create serious harms, as some people are exposed to hate that is left undetected while others are restricted in their freedom of expression when innocuous content is removed. These issues are exacerbated by biases in model performance, where hate against some groups is detected less reliably than hate against other groups.
Our project, HateCheck, is a multilingual auditing tool for hate speech detection models. Typically, these models have been tested on held-out test sets using summary metrics like accuracy. This makes it difficult to pinpoint model weak spots. Instead, HateCheck introduces functional tests for hate speech detection models, to enable more targeted diagnostics of potential model weaknesses and biases. Each functional test contains test cases for different types of hate and challenging non-hate, including special cases for emoji-based hate. In total, HateCheck has 36,000+ test cases across 11 languages, with further expansions planned.
Our work on HateCheck has been published at top NLP conferences, and covered by major media outlets. HateCheck is a substantial innovation in how hate speech detection models are evaluated, which supports the creation of fairer and more effective hate speech detection.
For more info, visit www.hatecheck.ai
Name Affiliation Paul Röttger University of Oxford Hannah Rose Kirk University of Oxford Bertie Vidgen University of Oxford We present a tool that allows people to easily evaluate computer generated predictions for fairness. The tool is an interactive, downloadable package that takes the user through the process of testing for biased predictions. The user can make allowances for expected differences in results, an example for when this would be required is throat cancer which is more common in men. Additionally, the tool allows for training data to be tested, to understand whether there is an unknown imbalance that the computer is trying to replicate.
It has been tested using two distinct prediction generators and datasets. The first is Surgical Outcome Risk Tool (SORT), which predicts a patient's risk of death within 30 days of inpatient surgery. There are 7 inputs to SORT including the clinician’s subjective risk assessment and returns a percentage mortality risk. SORT is in active use in the UK to support the consent to surgery process, as well as for decisions on resource allocation including critical care admission after surgery. The tool shows that SORT is fair, with regards to patient sex and statistical parity. That is, groups have an equal chance of a positive result.
The second is a synthetic dataset and job hiring predictions, that includes length of work, length of study and gender. The tool shows that the predictions are not fair with regards to sex but match the original dataset well, implying that the issue may be in the dataset rather than the predictions. The use of this dataset shows that our tool is scalable to areas and models other than SORT.
Name Affiliation Siân Carey (Lead) University of Leeds Alwyn Kotzé Leeds Teaching Hospitals NHS Trust Ciarán McInerney University of Sheffield Tom Lawton Bradford Teaching Hospitals NHS Foundation Trust Ibrahim Habli University of York Owen Johnson University of Leeds Marc de Kamps University of Leeds The Joint Fairness Assessment Method developed by NGO Algorithm Audit combines data-driven bias testing with normative and context-sensitive judgment of human experts, to determine fair AI on a case-by-case basis. The data-driven component comprises an unsupervised clustering tool (available as a free-to-use web application) that discovers complex and hidden forms of bias. It thereby tackles the difficult problem of detecting proxy-discrimination that stems from unforeseen and higher-dimensional forms of bias, including intersectional forms of discrimination. The results of the bias scan tool serve as a starting point for a deliberative assessment by human experts to evaluate potential discrimination and unfairness in an AI system.
As an example, we applied our bias scan tool to a BERT-based disinformation classifier and distilled a set of pressing questions about its performance and possible biases. We presented these questions to an independent audit commission composed of four academic experts on fair AI, and two civil society organizations working on disinformation detection. The audit commission believes there is a low risk of (higher-dimensional) proxy discrimination by the reviewed disinformation classifier. The commission judged that the differences in treatment identified by the quantitative bias scan can be justified, if certain conditions apply.
Our joint approach to fair AI is supported by 20+ actors from the AI auditing community, including journalists, civil society organizations, NGOs, corporate data scientists and academics. In sum, it combines the power of rigorous, machine learning-informed bias testing with the balanced judgment of human experts, to determine fair AI in a concrete way.
Name Affiliation Jurriaan Parie Algorithm Audit Ariën Voogt Algorithm Audit Joel Persson Algorithm Audit Applications of probabilistic models are increasingly used as a part of AI systems shaping the fabric of our society. These AI systems can exhibit or exacerbate undesirable biases, cause disproportionate harm to underrepresented groups, or lead to potentially adverse risks. These potential negative impacts can be linked to the absence of proper auditing tools, lack of proper regulatory control, and misuse of AI models. It is paramount we evaluate performance of AI systems not only from a lens of predictive power and rate of error but also from a lens of trustworthiness. To address this challenge, we introduce KiTE. KiTE serves as an open-source solution to auditing and improving the trustworthiness of predictive models. KiTE is an open-sourced, Python package that allows the end-user to (1) hypothesis test the trustworthiness of a predictive model and (2) perform post-processing model calibration to achieve better trustworthiness. To improve the accessibility and usability of our auditing tools, we also provide a user-friendly web dashboard that allows end-users to (1) evaluate model calibration, (2) identify and quantify areas of group-wise bias, and (3) hypothesis test their model’s trustworthiness. We showcase an application of this auditing tool in criminal justice and provide several case studies and example data sets to illustrate the versatility of the proposed solution. We evaluate the trustworthiness of a binary classification model we created using COMPAS data. Adaptation of KiTE’s auditing toolset can lead to more just outcomes in various settings, including employment, health care, education, lending, and criminal justice.
Name Affiliation Prachi Ingle University of Texas at Austin Arya Farahi University of Texas at Austin LUCID: Language Model Co-auditing through Community-based Red Teaming is an AI bias assessment tool that aims to automate certain aspects of AI bias red teaming, foster a community-based approach to auditing AI systems, and provide a platform for documenting, identifying, and discussing instances of bias in text. The tool uses semi-automated prompt engineering and community-based red teaming to enhance bias detection over a range of large language models, with the goal of empowering marginalized communities to audit algorithmic decision-making tools themselves and hold companies accountable.
LUCID can generate texts that cover a wide variety of scenarios and personas and compare the outputs of multiple LLMs. Currently, the tool focuses on comparing protected categories such as race, gender, religion, and nationality and can be expanded to include language and occupation. LUCID uses OpenAI’s GPT-3 Davinci and Perspective API, but can easily integrate new APIs as they become available.
The script receives seed sentences through a Google form. It replaces words related to protected categories with alternative words to generate new sentences. Each new sentence is then analyzed for toxicity using Perspective API. The script also generates text completions for each new sentence using GPT-3, calculates the word count of each completion, and determines the "controversy scores" for each completion in the persona of male and female humans. Finally, the script writes the original sentences, their Perspective API score, prompt completion, word count, and controversy scores to a new Google Sheet so auditors can identify and document instances of bias. Read more here.
Name Affiliation Roya Pakzad Taraaz Automated decision-making systems (ADMs) are increasingly used to make consequential decisions that can significantly impact people's lives. This raises the risk of bias, particularly against marginalized communities. To address this problem, we developed the FairnessLab, a web application that allows users to audit ADMs for biases across socio-demographic groups. Unlike existing tools that often oversimplify the evaluation of bias and offer limited fairness metrics without sufficient guidance on which metrics to use, the FairnessLab builds on a more nuanced philosophical approach: It guides users in generating fairness metrics that are morally appropriate for their use case by taking the societal and legal context of the ADM into account. By grappling with what it means to be fair in different contexts, FairnessLab users can identify biases that other tools might overlook. This is essential for civil society members, journalists, and other stakeholders who want to ensure that ADMs are deployed in a fair and unbiased way. Visit our website to learn more about the FairnessLab and how you can use it to promote fairness and equity in automated decision-making.
Name Affiliation Corinna Hertweck (Co-Lead) University of Zurich & Zurich University of Applied Sciences Joachim Baumann (Co-Lead) University of Zurich & Zurich University of Applied Sciences Michele Loi Polytechnic University of Milan Christoph Heitz Zurich University of Applied Sciences Large language models (LLMs), which power products such as ChatGPT, have become increasingly popular for their ability to generate high-quality human-like reasoning and text. However, prior research has revealed that LLMs can generate toxic text and reflect human biases. These studies have demonstrated bias and toxicity towards particular groups, but have not offered a comprehensive solution that quantifies toxicity bias across a large number of communities and identities.
Tobias is an interactive tool that can help regulators, developers, and the public better understand LLMs and explore levels of toxicity and toxicity bias towards specific groups. Importantly, Tobias measures toxicity and toxicity bias scores across a wide range of identities. Through a toxicity bias score, Tobias captures the extent to which a model is more biased towards certain groups compared to a baseline toxicity level. In addition, Tobias also helps explore implicit biases—the ways in which an LLM might “talk” about or generate toxic texts towards a certain group—by providing insight into the themes that underlie a model’s generated text.
Tobias builds on numerous projects: it leverages Facebook’s HolisticBias dataset to generate prompts across many communities and identities, Unity.ai’s Detoxify and Google Jigsaw’s Perspective API to measure toxicity scores, and BERTopic to automatically extract topics for text generated by an LLM. Tobias has been pre-loaded with several models for efficiency and to reduce environmental burden in exploring toxicity and toxicity bias for both (1) open-source models (e.g., GPT-2) using the HuggingFace library and (2) OpenAI’s deployed models, including the engine underlying ChatGPT. Tobias will be released as open source in the future and support the addition of custom models.
Tobias can be found at: https://tobias.keystonestrategy.io/.
*Acknowledgements. We want to share our special thanks to Professor Vincent Conitzer at Carnegie Mellon University for his invaluable advice in the development of Tobias.
Name Affiliation Junsu Choi Keystone Strategy Sydney Cohen Keystone Strategy Aaron White Keystone Strategy Bohan Li Keystone Strategy Nicholas Goutermout Keystone Strategy Kilbourne Charrier Keystone Strategy Prithvi Punjabi Keystone Strategy Carley Reardon Keystone Strategy Rohit Chatterjee Keystone Strategy Zeno is an interactive platform for auditing AI systems across diverse use cases to discover systematic failures and biases. Our goal with Zeno is to build the go-to, open-source platform that both model developers and non-technical stakeholders, such as policymakers and regulators, can use to understand and evaluate model behavior. To design Zeno, we conducted 18 need-finding interviews with practitioners and ML auditors to understand their current pain points for auditing and the limitations of existing tools.
The Zeno platform consists of an extensible Python API used to scaffold a user interface (UI) for interactively evaluating models. The Python API sets up Zeno with model predictions, metric functions, and dataset augmentations. Nontechnical users can then interact with the UI to explore the performance of models on subsets of data, define formal slices, and export reports and visualizations. Zeno can audit any Python-based model since the API and UI are model-, task-, and data-type agnostic.
Zeno has been applied to audit a wide range of models across data modalities and model architectures. We have found that Zeno's abstractions empower technical and non-technical users to find significant biases in models ranging from audio transcription to image generation. Zeno provides an accessible yet powerful platform that diverse stakeholders can use to interactively and systematically audit the vast array of AI systems impacting our daily lives.
For more information, please visit https://zenoml.com/ and https://github.com/zeno-ml/zeno.
Name Affiliation Ángel Alexander Cabrera (Lead) Carnegie Mellon University Donald Bertucci Oregon State University Erica Fu Carnegie Mellon University Kenneth Holstein Carnegie Mellon University Ameet Talwalkar Carnegie Mellon University Adam Perer Carnegie Mellon University Jason I. Hong Carnegie Mellon University
Contact Us
For questions, please contact us at algorithmicaudits@stanford.edu.