Day 2 Agenda: The First Workshop of a Public AI Assistant to World Wide Knowledge (WWK)

Back to Summit Overview Day 1 Agenda

February 14, 2025

Tutorial: Transforming LLMs into Reliable Knowledge Assistants

Large Language Models (LLMs) offer incredible potential for information retrieval but can sometimes produce inaccurate details. This tutorial will introduce advanced techniques for building virtual assistants that effectively navigate and analyze diverse knowledge sources, including free-text documents, structured databases, knowledge graphs, and even printed text.

You'll see demonstrations of Stanford's Genie technology, showcasing how it writes comprehensive articles by researching the internet for over 300,000 consumers, helps historians study the 19th-century African Times corpus, assists journalists with Federal Election Commission data, and enables users to extract information from Wikidata, the world's largest knowledge graph. 

Discover how to harness LLMs to create trustworthy and efficient knowledge assistants for various informational needs on your own knowledge corpus.

8:30 am – 9:00 am | Breakfast

9:00 am - 9:10 am | Overview of the Genie knowledge assistant 

  • Monica Lam

The Genie assistant is designed to work with any corpus of data, consisting of 

  • Images of printed material
  • Free text
  • Databases
  • Knowledge graphs

Knowledge access functions include: 

  • Browse
  • Semantic search
  • Interactive chat with near-zero hallucination
  • Answer queries for both free-text and structured data.
  • Write comprehensive research articles with fine-grained citations

 Genie can also

  • Perform interactive tasks under developer control
  • Extract knowledge graph from a document set

  • Detect inconsistencies in existing corpora

9:10 am -  9:40 am | Exploring a Corpus: From Physical Paper to Reliable Multilingual Chatbots

  • Sina Semnani

Given a free-text corpus of any size and in any language (supported by LLMs), Genie automatically allows users to browse, search, and chat with the corpus. To minimize hallucinations, Genie employs a hybrid Retrieval-Augmented Generation (RAG) strategy, where LLM-generated text is filtered claim-by-claim against the corpus and integrated into an enhanced retrieval pipeline.

Genie can handle historical newspaper scans with complex page layouts where an article is scattered across different columns and pages.

Case Studies:

  • Wikipedia in 25 languages: Genie achieves 98% factual accuracy in conversations with users about recent topics, compared to 43% with GPT-4. This research won the Wikimedia Foundation Research Award of the Year. (https://wikichat.genie.stanford.edu, https://search.genie.stanford.edu)
  • The African Times: A newspaper published in the late 19th century by the African diaspora (obtained from The British Museum). (https://history.genie.stanford.edu)
  • Chronicling America: A collection of over 200 years of American historical newspapers (obtained from the Library of Congress).

Reference: https://arxiv.org/abs/2305.14292

9:40 am - 10:10 am | Writing Research Articles with Full Citations

  • Yucheng Jiang

Genie can take any topic and write a full article from a given corpus, complete with references. Genie simulates users with different perspectives to ask pertinent questions on the topic, retrieve information, review it, and ask further questions. Users can interact with the experts in a roundtable discussion to explore their topics of interest; this new paradigm is shown to be much more effective than simple question answering.     

Case studies: An internet subset (as approved by Wikipedia); Semantic Scholar. Over 300,000 users have requested 500,000 topics from lifestyles to scientific research. (https://storm.genie.stanford.edu)

References: https://arxiv.org/abs/2402.14207; https://arxiv.org/abs/2408.15232

10:10 am - 10:40 am | Answering Queries of Structured and Unstructured Data

  • Shicheng Liu

Knowledge corpora often consists of a combination of free text and structured data. When given free-text documents and schemas for the structured data, Genie accepts natural language queries and automatically retrieves the data from the hybrid data sources. Genie translates users’ natural language requests  into formal queries involving tabular, graph, and hybrid databases. It leverages SQL, SPARQL, and the novel SUQL language, which extends SQL to integrate information retrieval for free text with relational database concepts. The Genie agent uses an agentic approach that combines evaluation of intermediate queries and navigation of the knowledge graph to generate the final query.

Case studies:

References: https://arxiv.org/abs/2407.11417; https://arxiv.org/abs/2311.09818.

10:40 am - 11:00 am | Break

11:00 - 11:30 | Performing Interactive Tasks under Developer Control

  • Harshit Joshi

Many interactive tasks, from customer support to business operations, require agents that can provide accurate knowledge and perform specific functions;  direct prompting of LLMs tends to fail on uncommon inputs and complex tasks. Genie combines LLMs with a new Genie Worksheet system that gives developers full control over the agent actions. 

Case studies: Course advisor, Writing research grants (https://ws.genie.stanford.edu/)

Reference: https://arxiv.org/abs/2407.05674

11:30 am - 11:45 am | Extracting Structured Knowledge from Text

  • Sina Semnani

Studying trends in aggregate often requires access to structured data, such as databases or knowledge graphs. For instance, monitoring disease outbreaks or political violence becomes more manageable when we have access to the "what," "who," "when," and "where" of events of interest. However, this type of data is frequently unavailable.

Genie addresses this challenge by automatically extracting the necessary information from an unstructured corpus, based on user-defined requirements. Unlike other existing tools, Genie supports long documents, multiple languages, and user-defined domain-specific entities.

Case study: Armed Conflict Location and Event Data (ACLED)

11:45 am - 12:00 pm | Inconsistency Detection in Large Corpora

  • Sina Semnani

Many knowledge corpora, such as company knowledge bases, software documentation, and open knowledge sources like Wikipedia, are manually curated over extended periods by multiple contributors. This curation process can lead to internal inconsistencies within the corpora, such as when new information is updated in one section but not in others. Genie tackles this problem by automatically identifying potential inconsistencies using a Large Language Model agent and presenting them for human review.

Case Study: Wikipedia: Our preliminary experiments indicate that there are about three inconsistent facts in every ten paragraphs of the English Wikipedia.