Common Crawl Foundation | Preserving Humanity's Knowledge and Making it Accessible: Addressing Challenges of Public Web Data
Learn about Common Crawl's insights from a recent data product and informed solutions for the future of public web data.
The Common Crawl Foundation is dedicated to preserving humanity's knowledge and making it accessible through its free public web dataset, a vital resource since 2008. As AI development accelerates, concerns have emerged regarding the accessibility and transparency of public web data, impacting open datasets in three key ways: robots.txt exclusions, legal demands, and "bot defenses." Two of these are not visible in public and are not very well understood. In this seminar, Common Crawl will present insights from a new data product that utilizes Common Crawl's crawl metadata to visually explore these three problems, advocating for greater transparency and informed solutions for the future of public web data.
Related Events
AI+Education Summit 2026
The AI Inflection Point: What, How, and Why We Learn
Tom Mitchell | The History of Machine Learning
How did we get to today’s technology which now supports a trillion dollar AI industry? What were the key scientific breakthroughs? What were the surprises and dead-ends along the way...
How did we get to today’s technology which now supports a trillion dollar AI industry? What were the key scientific breakthroughs? What were the surprises and dead-ends along the way...
Gaidi Faraj, Lofred Madzou | Nurturing Africa’s AI Leaders through Math Olympiad
The African Olympiad Academy is a world-class high school dedicated to training Africa’s most promising students in mathematics, science, and artificial intelligence through olympiad-based pedagogy.
The African Olympiad Academy is a world-class high school dedicated to training Africa’s most promising students in mathematics, science, and artificial intelligence through olympiad-based pedagogy.