AI Researchers Tap into Medical Twitter To Create Powerful New Analysis Tool

Most are familiar with Twitter — recently redubbed X — but few are aware that this popular social media channel also plays host to various sub-communities. Take medical Twitter, for instance, where an energetic community of clinicians trades interesting, intriguing, and often perplexing medical images and asks other highly trained clinicians to weigh in.

Recognizing a rich, untapped resource, researchers at Stanford University have combed medical Twitter to gather more than 200,000 anonymized diagnostic images of cancers and other diseases, along with the professional annotations provided by those highly trained clinicians. From that data, they have built a powerful artificial intelligence (AI) model able to correctly analyze and diagnose previously unseen images.

“One of the biggest challenges of developing diagnostic AI is lack of large-scale annotated data,”  says James Zou, a professor of biomedical data science and member of the Stanford Institute for Human-Centered AI (HAI). “But right there was this highly trained group of physicians sharing data and insights on social media. Medical Twitter is a tremendous boon to medical AI.”

Read the preprint study, "Leveraging Medical Twitter to Build a Visual–language Foundation Model for Pathology AI"

Despite the clear promise, Zou says, computational pathology, as the field is known, has lagged behind other areas of AI due to the dearth of anonymized, well-annotated image datasets, complicated by the fact that there is a universe of 8,000 known diseases to classify — and an untold number of potentially valuable images kept locked away in private hospital databases.

In paging through medical Twitter, Zhi Huang and Federico Bianchi, postdocs at Stanford and co-leaders of the project, realized that with each posted image, each insightful analysis, each careful hashtag (#breastcancer), and each probing question, community members were in effect labeling the data.

“A pathologist would run up against something they’d never seen before and post the image to the community, asking, ‘What do you think is going on here?’ And a knowledgeable group of colleagues around the world would respond in written text,” Zou explains. “In that combination of text and images, we had our resource.”

Gathering and curating more than 243,000 diagnostic images and their comments from medical Twitter, anonymizing them and winnowing out the poor-quality images and responses, Huang, Bianchi, and the team created OpenPath, one of the largest public pathology datasets with natural language descriptions. The team then trained PLIP — a foundation AI that can understand both images and text.

“That pairing of images and text makes it quite useful. PLIP enables researchers to retrieve similar cases by searching using either images or words,” Zou says. “We think it should greatly facilitate knowledge sharing within the pathology community worldwide.”

Asked about some of the recent changes at Twitter — including potential changes in the API that restrict researchers from gathering this kind of data — Zou says he’s not troubled. The real message in this research is not specific to Twitter, but something larger, more democratic — and more exciting.

“The insight here may be that much medical knowledge is shared on social media,” Zou says. “I think we can be very creative in looking for new and diverse sources of data to improve medical AI.”

Stanford HAI’s mission is to advance AI research, education, policy and practice to improve the human condition. Learn more.