The prevalence of mental illness, which affects nearly one in five American adults, has created a spike in unmet demand. Peer support, in which volunteers offer immediate help through empathetic listening, has emerged as a promising complement to professional treatment.
While this informal approach allows for immediate and accessible support, it also presents unique challenges in preparing supporters for their role. Unlike the structured training programs found in professions such as psychotherapy or medicine, peer counselors often start out with far less preparation. To harness the full potential of peer support, it's crucial to equip volunteers with robust resources and ongoing development opportunities.
In a new working paper accepted in the 2024 Association for Computational Linguistics conference, researchers from Stanford, Carnegie Mellon, and Georgia Tech present an AI-based model that offers feedback to novice peer counselors to improve their ability to help others in need. The project is rooted in a unique partnership between Stanford computer scientist Diyi Yang and Stanford psychologist Bruce Arnow, who are both authors on the paper, which received support from the Stanford Institute for Human-Centered AI.
“Interdisciplinary collaboration was essential in undertaking this project,” Arnow says. “AI has enormous potential to help improve both the quality and efficiency of psychotherapy training, but the mental health community is not well equipped to develop an AI-assisted training model and the computer science community is not grounded in counseling intervention skills. Forming a team with both disciplines enabled us to progress in this exciting new area of investigation.”
Defining Good Feedback
The first step in the project was mapping out what constitutes helpful feedback and how to offer it to new peer counselors. The researchers worked with three psychotherapists at Stanford to develop a realistic and practical blueprint of good feedback. “The goal was to think about what kind of mistakes novice peer counselors often make and what kind of advice can help them learn,” says Alicja Chaszczewicz, a PhD student in computer science at Stanford and co-author on the paper.
By working with counseling supervisors, the research team developed a framework to provide three pieces of information for any given exchange: a clear definition of what the counselor should be trying to understand from the conversation; suggestions to improve the counselor’s response (more empathy, more professionalism, better questions, etc.); and a specific suggested response that would best align with the concerns being expressed and the goal of the conversation. The model can also give positive reinforcement if counselors give a good response.
With this feedback framework in mind, the researchers then collected a dataset of feedback given in 400 different emotional support conversations. Each utterance in the dataset received co-annotated feedback: GPT-4 wrote a first draft and two domain-experts decided on final edits. This dataset provided a high-quality ground truth upon which to fine-tune a model to automatically generate feedback that mirrors what counseling supervisors would provide their trainees.
“The main objective was to make sure we minimized poor feedback from the model,” says Ryan Louie, a postdoctoral researcher at Stanford and co-author on the paper. To ensure this, they created an innovative self-checking process in which the model feeds its proposed feedback into its own framework to confirm whether the suggestion aligns with the conversation’s goals. “It was basically double-checking itself in order to mitigate the possibility it was giving poor advice.”
The resulting model, according to human experts who reviewed its output, is a valuable tool in coaching peer counselors who don’t have much formal training.
Training Counselors Using the Feedback Model
As the team explores how this feedback model can empower helpers-in-training, their aim is to target environments where direct, individualized supervision may be limited. A promising initial application could be in educational settings where instructors face challenges in providing detailed supervision on every counseling conversation. The feedback model could significantly enhance this experience by offering reminders of the goal of a specific part of the conversation and how a counselor's response could better align with the goal. Such detailed feedback can offer an additional perspective, complementing the periodic discussions classmates and instructors have with each other.
Ideally, however, the feedback model will be of use in environments without as much direct support.
One possibility the researchers mentioned is a training environment in which novice peer counselors take part in practice conversation with AI patients — and then these conversations are reviewed by the model. This would be a “safe sandbox,” in Louie’s words, where novice counselors could practice on their own and receive feedback on their counseling skills while sidestepping privacy concerns — “you may not want or be able to analyze the data of real patients,” Louie says. It would also allow counselors to experiment, make mistakes, and ultimately become more prepared before they are responsible for helping real people in need.
But the broader goal is to have this tool available to use at scale and offer trainees another learning resource.
“We are by no means attempting to replace the clinical supervision process, which is very complex,” Chaszczewicz says. “We’re trying to mimic this one part of it and, that way, provide both a pedagogical tool and a practical tool to support organizations that don’t have enough instructors to give their counselors feedback.”
Stanford HAI’s mission is to advance AI research, education, policy and practice to improve the human condition. Learn more.