Common Crawl Foundation | Preserving Humanity's Knowledge and Making it Accessible: Addressing Challenges of Public Web Data
Learn about Common Crawl's insights from a recent data product and informed solutions for the future of public web data.
Get the latest news, advances in research, policy work, and education program updates from HAI in your inbox weekly.
Sign Up For Latest News
Learn about Common Crawl's insights from a recent data product and informed solutions for the future of public web data.
In the last year, AI models have rapidly improved on public benchmarks measuring their ability to carry out research tasks autonomously. Such performance suggests that models may be capable of automating parts of the AI R&D process itself, thereby accelerating the pace of AI progress.

In the last year, AI models have rapidly improved on public benchmarks measuring their ability to carry out research tasks autonomously. Such performance suggests that models may be capable of automating parts of the AI R&D process itself, thereby accelerating the pace of AI progress.
This talk will cover key insights from the third AI100 report, AI Goes Mainstream: The Scaling Era and the Rise of Generative AI, set for release in November 2026. The report was commissioned by the AI100 standing committee and led by study panel chair Michael Wooldridge.

This talk will cover key insights from the third AI100 report, AI Goes Mainstream: The Scaling Era and the Rise of Generative AI, set for release in November 2026. The report was commissioned by the AI100 standing committee and led by study panel chair Michael Wooldridge.
Sessions run Wednesdays from 4:30–5:30 PM in CoDa E160. Each session features a different Stanford speaker; talk titles are announced by the organizers.

Sessions run Wednesdays from 4:30–5:30 PM in CoDa E160. Each session features a different Stanford speaker; talk titles are announced by the organizers.
The Common Crawl Foundation is dedicated to preserving humanity's knowledge and making it accessible through its free public web dataset, a vital resource since 2008. As AI development accelerates, concerns have emerged regarding the accessibility and transparency of public web data, impacting open datasets in three key ways: robots.txt exclusions, legal demands, and "bot defenses." Two of these are not visible in public and are not very well understood. In this seminar, Common Crawl will present insights from a new data product that utilizes Common Crawl's crawl metadata to visually explore these three problems, advocating for greater transparency and informed solutions for the future of public web data.