Stanford
University
  • Stanford Home
  • Maps & Directions
  • Search Stanford
  • Emergency Info
  • Terms of Use
  • Privacy
  • Copyright
  • Trademarks
  • Non-Discrimination
  • Accessibility
© Stanford University.  Stanford, California 94305.
AI Researchers Tap into Medical Twitter To Create Powerful New Analysis Tool | Stanford HAI

Stay Up To Date

Get the latest news, advances in research, policy work, and education program updates from HAI in your inbox weekly.

Sign Up For Latest News

Navigate
  • About
  • Events
  • Careers
  • Search
Participate
  • Get Involved
  • Support HAI
  • Contact Us
Skip to content
  • About

    • About
    • People
    • Get Involved with HAI
    • Support HAI
  • Research

    • Research
    • Fellowship Programs
    • Grants
    • Student Affinity Groups
    • Centers & Labs
    • Research Publications
    • Research Partners
  • Education

    • Education
    • Executive and Professional Education
    • Government and Policymakers
    • K-12
    • Stanford Students
  • Policy

    • Policy
    • Policy Publications
    • Policymaker Education
    • Student Opportunities
  • AI Index

    • AI Index
    • AI Index Report
    • Global Vibrancy Tool
    • People
  • News
  • Events
  • Industry
  • Centers & Labs
news

AI Researchers Tap into Medical Twitter To Create Powerful New Analysis Tool

Date
August 28, 2023
Topics
Healthcare
Communications, Media
DALL-E

Stanford researchers discover a rich new data source in the anonymized pathology images and online comments of thousands of pathologists.

Most are familiar with Twitter — recently redubbed X — but few are aware that this popular social media channel also plays host to various sub-communities. Take medical Twitter, for instance, where an energetic community of clinicians trades interesting, intriguing, and often perplexing medical images and asks other highly trained clinicians to weigh in.

Recognizing a rich, untapped resource, researchers at Stanford University have combed medical Twitter to gather more than 200,000 anonymized diagnostic images of cancers and other diseases, along with the professional annotations provided by those highly trained clinicians. From that data, they have built a powerful artificial intelligence (AI) model able to correctly analyze and diagnose previously unseen images.

“One of the biggest challenges of developing diagnostic AI is lack of large-scale annotated data,”  says James Zou, a professor of biomedical data science and member of the Stanford Institute for Human-Centered AI (HAI). “But right there was this highly trained group of physicians sharing data and insights on social media. Medical Twitter is a tremendous boon to medical AI.”

Read the preprint study, "Leveraging Medical Twitter to Build a Visual–language Foundation Model for Pathology AI"

 

Despite the clear promise, Zou says, computational pathology, as the field is known, has lagged behind other areas of AI due to the dearth of anonymized, well-annotated image datasets, complicated by the fact that there is a universe of 8,000 known diseases to classify — and an untold number of potentially valuable images kept locked away in private hospital databases.

In paging through medical Twitter, Zhi Huang and Federico Bianchi, postdocs at Stanford and co-leaders of the project, realized that with each posted image, each insightful analysis, each careful hashtag (#breastcancer), and each probing question, community members were in effect labeling the data.

“A pathologist would run up against something they’d never seen before and post the image to the community, asking, ‘What do you think is going on here?’ And a knowledgeable group of colleagues around the world would respond in written text,” Zou explains. “In that combination of text and images, we had our resource.”

Gathering and curating more than 243,000 diagnostic images and their comments from medical Twitter, anonymizing them and winnowing out the poor-quality images and responses, Huang, Bianchi, and the team created OpenPath, one of the largest public pathology datasets with natural language descriptions. The team then trained PLIP — a foundation AI that can understand both images and text.

“That pairing of images and text makes it quite useful. PLIP enables researchers to retrieve similar cases by searching using either images or words,” Zou says. “We think it should greatly facilitate knowledge sharing within the pathology community worldwide.”

Asked about some of the recent changes at Twitter — including potential changes in the API that restrict researchers from gathering this kind of data — Zou says he’s not troubled. The real message in this research is not specific to Twitter, but something larger, more democratic — and more exciting.

“The insight here may be that much medical knowledge is shared on social media,” Zou says. “I think we can be very creative in looking for new and diverse sources of data to improve medical AI.”

Stanford HAI’s mission is to advance AI research, education, policy and practice to improve the human condition. Learn more. 

Share
Link copied to clipboard!
Contributor(s)
Andrew Myers

Related News

Ambient Intelligence, Human Impact
May 07, 2025
News

Health care providers struggle to catch early signals of cognitive decline. AI and computational neuroscientist Ehsan Adeli’s innovative computer vision tools may offer a solution.

News

Ambient Intelligence, Human Impact

HealthcareComputer VisionMay 07

Health care providers struggle to catch early signals of cognitive decline. AI and computational neuroscientist Ehsan Adeli’s innovative computer vision tools may offer a solution.

MedArena: Comparing LLMs for Medicine in the Wild
Eric Wu, Kevin Wu, James Zou
Apr 24, 2025
News

Stanford scholars leverage physicians to evaluate 11 large language models in real-world settings.

News

MedArena: Comparing LLMs for Medicine in the Wild

Eric Wu, Kevin Wu, James Zou
HealthcareNatural Language ProcessingGenerative AIApr 24

Stanford scholars leverage physicians to evaluate 11 large language models in real-world settings.

AI in Science and Medicine: A Deep Dive from the AI Index Report
Hanae Armitage
Apr 15, 2025
News

HAI Associate Director Russ Altman discusses how artificial intelligence improves patient care and expands research capacity, as reported by the AI Index from the Institute for Human-Centered Artificial Intelligence.

News

AI in Science and Medicine: A Deep Dive from the AI Index Report

Hanae Armitage
HealthcareApr 15

HAI Associate Director Russ Altman discusses how artificial intelligence improves patient care and expands research capacity, as reported by the AI Index from the Institute for Human-Centered Artificial Intelligence.