AI Agents Simulate 1,052 Individuals’ Personalities with Impressive Accuracy

It’s no longer science fiction: Your personality – your beliefs, quirks, and decision-making patterns – can be captured and brought to life inside an artificial mind. 

Stanford researchers have simulated the personalities of 1,052 individuals with impressive accuracy using interviews and a large language model (LLM). These virtual agents exhibit personas that answer questions and make decisions in ways that mirror their real-life counterparts.

“It seems quite amazing that we could create these open-ended agents of real people,” says Joon Sung Park, a computer science graduate student at Stanford and lead researcher on the project. But he likens it to a good friend or perhaps a therapist being able to predict how a particular person would answer the questions.

“The language model is trying to role-play as the person it just interviewed,” he says. In addition to the interview script, the agent relies on all of the psychological and social science expertise that is embedded in the LLM. “It’s a very powerful combination.”

While Park notes that the work, published on preprint server ArXiv, might provoke entirely reasonable concerns about deepfake videos, co-option of individuals’ likenesses, and a world where people have conversations with AI versions of their friends or relatives, he and his team have put guardrails in place to forestall such uses.

For this research team, which includes HAI affiliate faculty members Michael Bernstein, Robb Willer, and Percy Liang, as well as faculty and staff from Northwestern University, the University of Washington, and Google DeepMind, the value of these agents is laudable: to create a realistic population of generative agents to use as a testbed to study impacts of policy proposals, from solving the climate crisis to preventing the next global pandemic. 

“We think these kinds of agents are going to power a lot of future policymaking and science,” Park says.

From Believability to Accuracy

A year ago, Park and his colleagues took a first stab at creating believable generative AI agents. They assigned a paragraph-long fictional biography to each of 25 generative AI agents and set them loose in a virtual landscape. As described in a previous HAI article, the agents interacted with one another in believable ways: They went to work, prepared meals, planned a party, and discussed who was running for mayor.

But reliable social science research needs more than believability, Park says. “If you’re designing a new policy, you want your simulations to be accurate.”

To achieve that goal, the researchers needed to create a large population of agents with true-life backstories. And the population needed to be representative of the U.S. population in terms of age, race, gender, ethnicity, education level, and political ideology. 

The team recruited and interviewed 1,052 study participants who met those criteria. And because the interviews had to be standardized (and 1,000+ interviews would require more time and energy than a single graduate student could provide), Park and his team created an AI interviewer.

The 2-hour interview asked participants the story of their lives and their views on controversial issues, with follow-up questions based on the individuals’ previous answers. Ultimately, the interview transcripts went into the computer memory for each of 1,052 generative agents.

In addition, the team asked an LLM to review the interview transcript and evaluate certain aspects of each interviewee’s personality from the perspective of a particular type of expert – such as a social psychologist, economist, or sociologist. So, for example, an economist might comment that a person is cautious or a risk-taker, while a social psychologist might remark on the person’s extroversion. “We’re basically asking the language model to create a higher-level synthesis to capture certain ideas about these individuals,” Park says. That synthesis was also added to the generative agents’ memories.

Testing Generative Agents’ Accuracy

To determine whether the study participants’ views and personalities had been accurately captured by the generative agents, both the participants and the agents were given four tasks: Answer questions about their opinions, behaviors, and attitudes using the General Social Survey (GSS); answer the 44-item Big Five Inventory, which is designed to assess an individual’s personality; complete five well-known behavioral economic games (including the dictator game, trust game, public goods game, and prisoner’s dilemma); and participate in five social science experiments.

“These are experiments that are canonical in the social sciences, and the participants’ answers basically become our ground truth,” Park says. “Our agents then try to predict what people said in each of these buckets.”

Participants did the tasks twice, two weeks apart, because people’s preferences often change. 

The agents’ accuracy was impressive: They matched the participants’ answers on the GSS 85% as accurately as participants matched their own answers two weeks later. The agents also did well on the personality tests (80% correlation) and the economic games (66% correlation).

The interview-based agents’ GSS answers were also more accurate than agents whose memories included only the participants’ demographic information or agents whose memories consisted of a short paragraph that each study participant wrote about themself. In fact, the researchers showed that the interview-based generative agents were not only more accurate than these alternatives but also less biased.

“The beauty of having interview data is that it includes people’s idiosyncrasies and therefore the language models don’t resort to making race-based generalizations as often,” Park says.

Risks and Benefits

The Stanford research team is appropriately concerned about misuse of generative agents. “As scientists, it’s important that we set the right social standards and protections around this,” Park says.

A generative agent can be thought of as a new way of taking a self-portrait that tells a rich story about who a person is, but it is still a computational entity, Park notes. As such, like our genomic data, it should belong to and be controlled by the person whose portrait it represents, he says.

Therefore, the generative agents are not being released for public use. Anyone wanting to use them for research purposes must apply for access and provide strict assurances of privacy protection.

In addition, the team established an audit log for the use of every agent. This allows individuals represented by agents to see what their agents are doing and to have complete control over them. Individuals also have the ability to pull their consent. “If you don’t want your data involved in this anymore, you should be able to remove your agent from all contexts where it is functioning,” Park says.

Park wants to make sure the right mechanisms are put in place to minimize the risk to the best of the team’s ability, but ultimately he thinks the potential benefits of accurate generative agents outweigh the risks.

“I really do think there are many societal problems we’re failing to address right now that could be made easier with this testbed.” 

In an early test of whether generative agents could function as such a testbed, the research team tasked their AI agents with replicating the results of five different social science research projects. And, like the study participants themselves, the agents replicated four of the five studies. “It’s a great sign that we can already support these concrete use cases,” Park says.

“Wicked problems like climate change and pandemic policies require very complex planning and reasoning about contingencies,” he says. “Simulation is one way we may finally have a chance to crack some of them open. The potential benefit is game-changing.”