Gather Olympic Games 2020 Data from Wikipedia
Collect and section 713 Wikipedia pages on the 2020 Olympics as context data for a confabulation-resistant Q&A model.
Why it matters
Collect and process recent data on the Olympic Games 2020 from Wikipedia to build a robust question-answering model. This asset focuses on gathering and structuring information that GPT-3 may not have encountered during pre-training.
Outcomes
What it gets done
Download 713 unique Wikipedia pages related to the Olympic Games 2020.
Filter out sections with fewer than 40 tokens to ensure sufficient context.
Split data into individual sections for use as context in subsequent QA tasks.
Save the processed section dataset for use in the next stage of the QA model.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-olympics-1-collect-data | bash Steps
Steps in the chain
Overview
1. Collect Wikipedia data about Olympic Games 2020
The first of three legacy notebooks collecting and sectioning 713 Wikipedia pages on the 2020 Olympics as training context for a confabulation-resistant question-answering model. Use it as historical reference for pre-embeddings Q&A training data collection. The project recommends its newer embeddings-based Question Answering cookbook for current work instead.
What it does
This is the first of three legacy notebooks (the project itself flags reliance on deprecated API endpoints and recommends the newer embeddings-based Q&A approach instead) building a question-answering model that only answers when a question's context genuinely supports an answer, rather than confabulating a plausible-sounding one. This notebook collects the underlying data: 713 unique Wikipedia pages about the 2020 Olympic Games (actually held in summer 2021, after GPT-3's training cutoff), split into individual sections that later serve as the context paragraphs for generated questions and answers.
When to use - and when NOT to
This is a legacy, three-notebook pipeline rather than a current recommendation - for a modern Q&A-over-documents approach, the project itself points to the newer embeddings-based Question Answering cookbook instead. Treat this specifically as historical reference for how confabulation-resistant Q&A training data was built before that approach existed: this notebook only handles data collection, notebook two generates question/answer pairs with Davinci-instruct from each section, and notebook three adds adversarial question/context mismatches (trained to answer "No sufficient context for answering the question") plus a discriminator model that predicts whether a question is answerable from a given context.
Inputs and outputs
Data extraction via the Wikipedia API takes roughly half an hour, with a similar amount of time for processing. Pages are filtered and split into sections by heading, discarding sections unlikely to hold real textual content and enforcing a per-section token limit; sections under 40 tokens are dropped outright since they're unlikely to carry enough context for a good question. The resulting section dataset is saved for the next notebook in the series.
Integrations
An optional exploration step notes that the topic actually spans both Winter and Summer Olympics 2020 - the dataset deliberately keeps that ambiguity and noise rather than filtering to Summer only - and that most sections come in under 500 tokens.
Who it's for
Teams studying the older approach to building context-grounded, confabulation-resistant Q&A training data, before adopting the project's own recommended embeddings-based method for new work. The underlying motivation is worth understanding regardless of era: base GPT-3 models tend to do a good job answering when the answer is actually present in the given paragraph, but when it isn't, they'll often still attempt an answer rather than admitting they can't - and training on adversarial no-context examples was this pipeline's way of teaching the model to recognize and admit that limitation instead of confabulating a confident-sounding answer.
Source README
Note: To answer questions based on text documents, we recommend the procedure in Question Answering using Embeddings. Some of the code below may rely on deprecated API endpoints.
1. Collect Wikipedia data about Olympic Games 2020
The idea of this project is to create a question answering model, based on a few paragraphs of provided text. Base GPT-3 models do a good job at answering questions when the answer is contained within the paragraph, however if the answer isn't contained, the base models tend to try their best to answer anyway, often leading to confabulated answers.
To create a model which answers questions only if there is sufficient context for doing so, we first create a dataset of questions and answers based on paragraphs of text. In order to train the model to answer only when the answer is present, we also add adversarial examples, where the question doesn't match the context. In those cases, we ask the model to output "No sufficient context for answering the question".
We will perform this task in three notebooks:
- The first (this) notebook focuses on collecting recent data, which GPT-3 didn't see during its pre-training. We picked the topic of Olympic Games 2020 (which actually took place in the summer of 2021), and downloaded 713 unique pages. We organized the dataset by individual sections, which will serve as context for asking and answering the questions.
- The second notebook will utilize Davinci-instruct to ask a few questions based on a Wikipedia section, as well as answer those questions, based on that section.
- The third notebook will utilize the dataset of context, question and answer pairs to additionally create adversarial questions and context pairs, where the question was not generated on that context. In those cases the model will be prompted to answer "No sufficient context for answering the question". We will also train a discriminator model, which predicts whether the question can be answered based on the context or not.
1.1 Data extraction using the wikipedia API
Extracting the data will take about half an hour, and processing will likely take about as much.
1.2 Filtering the Wikipedia pages and splitting them into sections by headings
We remove sections unlikely to contain textual information, and ensure that each section is not longer than the token limit
1.2.1 We create a dataset and filter out any sections with fewer than 40 tokens, as those are unlikely to contain enough context to ask a good question.
Save the section dataset
We will save the section dataset, for the next notebook
1.3 (Optional) Exploring the data
There appear to be winter and summer Olympics 2020. We chose to leave a little ambiguity and noise in the dataset, even though we were interested only in the Summer Olympics 2020.
We can see that the majority of section are fairly short (less than 500 tokens).
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.