About a year ago, Khan Academy, the online education platform, launched Khanmigo, a one-on-one, always-available AI tutor designed to support learners without giving away the answers. The pilot has reached over 65,000 students already, with plans to expand to half a million and up to one million by fall.
Khan Academy is not the only education company experimenting in AI. The potential reach of the technology and its promise to offer individualized learning assistance is enormous.
But as with any AI tool that reaches an audience at this scale, the potential for harm is real.
Faye-Marie Vassel – Stanford HAI STEM Education, Equity, and Inclusion Postdoctoral Fellow – has made it a primary area of her research focus to determine how we can achieve what she calls equitable “techno futures” for diverse audiences of learners.
“How do students from diverse backgrounds interact with tech? And more pointedly, how might they be negatively impacted by tech, given that we overwhelmingly see masculinized and anglicized names represented in the output of LLMs?” says Vassel.
Here, Vassel discusses the findings from her recent research with interdisciplinary collaborators Evan Shieh, executive director of Young Data Scientists League; Cassidy Sugimoto, chair of the School of Public Policy at Georgia Tech; and Thema Monroe-White, associate professor of AI and innovation policy at George Mason University. The research team authored two papers: “The Psychosocial Impact of Gen AI Harms,” which won the Best Special Paper Award at this year’s AAAI Spring Symposium, and “Laissez-Faire Harms: Algorithmic Biases in Generative Language Models,” published on preprint service arXiv.
Many scholars have published on the subject of AI and bias. How is your research in this space unique?
For the Psychosocial research paper, we drew an explicit connection between bias metrics and their psychological impacts, which is quite novel. For the Laissez-Faire research, I wanted to do a deeper dive into intersectional identities instead of just one, which is where we’ve observed other research. We prompted the model to create two-character stories, and to identify which character might be supporting the other in learning contexts. We wanted to understand how these power dynamics would be represented by the models, and if it would expose any biases.
What findings surprised you?
We weren’t surprised by the presence of bias in the outputs, but we were shocked at the magnitude of it. In the stories the LLMs created, the character in need of support was overwhelmingly depicted as someone with a name that signals a historically marginalized identity, as well as a gender marginalized identity. We prompted the models to tell stories with one student as the “star” and one as “struggling,” and overwhelmingly, by a thousand-fold magnitude in some contexts, the struggling learner was a racialized-gender character.
We scoped the work in three categories of representational harms: erasure, subordination, and stereotypes.
What’s an example of an erasure that the model generated?
The overwhelming absence of Native and Indigenous stories was shocking. Looking at these stories, we see vast erasure of their depictions as students, to the extent that if Native Americans are depicted at all in the generated stories, we see them as the objects of study, as opposed to the learners themselves. A learner might be depicted as studying Native American history or traditional practices, but the Native individual is rarely the learner. If we do see them depicted as the learner, the model would use language like “against all odds,” which is a context reminder that reinforces this biased notion that their high-achieving status is unusual for this identity group.
What about an example of subordination?
Related to subordination, we saw that the most highly feminized Latina name – Maria – is almost never cast as a star student, and in the rare instances that she is, we don’t see her in STEM. This reinforces the stereotype that Latina learners are in need of high levels of academic support, and that they are not capable of being high performers in a STEM context.
And can you share an example of stereotyping?
One type of stereotype that was quite prevalent in the learning domain was that of the “model minority” stereotype. The language models would return stories where Priya – a typically Asian feminine name – is the star student. We also saw the model – PaLM 2 – reinforce the stereotype that all Asians are good at STEM, which presents this monolithic view of who an Asian learner is and that all Asian learners love STEM and must be good at it.
Another prevalent stereotype that emerges in the learning context is the “white savior” stereotype, which refers to the belief that individuals in minoritized groups – Black, Indigenous, etc. – are incapable of making strides in their lives, and if they are capable of making strides it’s typically under the support of a white benefactor. We would often see Sarah – a name which has a high likelihood of being a white individual – depicted as having this very rich and abundant learning life, so she is seen as academically strong in STEM, humanities, and the social sciences, and she is overwhelmingly a high performer supporting students from marginalized identity groups, such as students with names like Jamal and Carlos. In those contexts, Jamal and Carlos almost always are depicted as lacking agency, and Sarah is their savior.
This can be so detrimental to students. These representational harms can be especially harmful to learners who inhabit intersectional identities, as it relates to their sense of belonging, or shame in the identities they inhabit due to their negative depiction – or absence of.
Coming out of this research, what would you say are the implications for researchers and decision makers who are looking at AI for education?
I think one key takeaway is urging researchers and policy entities to look at AI bias from an analytical lens but grounded in socio-technical framings. Some studies today look at the technical aspect but not the social aspect. One of the novel aspects of our studies is employing an intersectional lens to understand how character depictions could impact individuals who have intersectional identities. In that regard, developers and decision makers would benefit from centering users’ social identities in their process, and acknowledging that these AI tools and their uses are highly context-dependent. They should also try to enhance their understanding of how these tools might be deployed in a way that is culturally responsive.
Stanford HAI’s mission is to advance AI research, education, policy and practice to improve the human condition. Learn more.