Embed Wikipedia Articles for Search
Prepare a Wikipedia article dataset for semantic search: collect, chunk, embed, and store as CSV.
Why it matters
Prepare and embed Wikipedia articles for efficient semantic search and question answering. This asset processes raw text into searchable embeddings, enabling advanced retrieval of information.
Outcomes
What it gets done
Download Wikipedia articles on a specific topic.
Clean and chunk articles into manageable sections.
Generate embeddings for each text chunk using the OpenAI API.
Store embeddings for subsequent retrieval and search.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-embeddingwikipediaarticlesforsearch | bash Steps
Steps in the chain
Overview
Embedding Wikipedia articles for search
An OpenAI Cookbook notebook that prepares a document dataset for semantic search by collecting, chunking, embedding, and storing text as a CSV, using Wikipedia articles as the worked example. Use it as a template for building an embedding-searchable dataset. For large datasets, use a vector database and the parallel request script instead of CSV and serial calls.
What it does
This notebook prepares a dataset of Wikipedia articles for semantic search, feeding the companion Question_answering_using_embeddings notebook. It follows a four-step procedure: collect a few hundred Wikipedia articles (the worked example uses 2022 Winter Olympics articles), chunk each into short, semi-self-contained sections, embed each chunk with the OpenAI API, and store the resulting embeddings in a CSV file.
When to use - and when NOT to
Use this as a template for turning a document collection into an embedding-searchable dataset, particularly when building a retrieval-augmented Q&A system. It's sized for a few thousand text chunks stored in a CSV; for large datasets, the notebook itself recommends a vector database instead of CSV storage, and a parallel request script to stay under API rate limits when embedding at scale.
Inputs and outputs
Chunking discards low-value sections (like External Links and Footnotes), strips reference tags and excess whitespace, splits each article into sections, and prepends titles and subtitles to each section's text so GPT retains context about where it came from. Sections longer than about 1,600 tokens get recursively split in half along paragraph boundaries where possible, since GPT can only process a limited amount of text at once. There's no single correct chunk size: longer sections carry more context but risk mixing topics and cost more per retrieval, while shorter sections are cheaper and let more distinct sections be retrieved (helping recall), though overlapping sections may be needed to avoid cutting an answer across a section boundary.
Integrations
Set OPENAI_API_KEY as an environment variable before running - the OpenAI library reads it automatically, so no separate configuration step is needed once it's set. Missing libraries can be installed with a plain pip install (or a notebook-cell !pip install), followed by restarting the notebook kernel so the newly installed package is actually picked up. For embedding a large number of chunks, the notebook points to a separate api_request_parallel_processor.py script that parallelizes API calls while throttling to stay within rate limits, rather than looping through requests serially.
Who it's for
Developers building a search or Q&A system over a document collection who need a concrete, tested chunking and embedding recipe rather than starting from a blank notebook. The specific 1,600-token cutoff and paragraph-boundary splitting shown here are one reasonable default, not a universal rule - the notebook is explicit that section-size choice is a genuine tradeoff to tune for your own corpus and retrieval needs, not something to copy blindly.
Source README
Embedding Wikipedia articles for search
This notebook shows how we prepared a dataset of Wikipedia articles for search, used in Question_answering_using_embeddings.ipynb.
Procedure:
- Prerequisites: Import libraries, set API key (if needed)
- Collect: We download a few hundred Wikipedia articles about the 2022 Olympics
- Chunk: Documents are split into short, semi-self-contained sections to be embedded
- Embed: Each section is embedded with the OpenAI API
- Store: Embeddings are saved in a CSV file (for large datasets, use a vector database)
0. Prerequisites
Import libraries
Install any missing libraries with pip install in your terminal. E.g.,
pip install openai
(You can also do this in a notebook cell with !pip install openai.)
If you install any libraries, be sure to restart the notebook kernel.
Set API key (if needed)
Note that the OpenAI library will try to read your API key from the OPENAI_API_KEY environment variable. If you haven't already, set this environment variable by following these instructions.
1. Collect documents
In this example, we'll download a few hundred Wikipedia articles related to the 2022 Winter Olympics.
2. Chunk documents
Now that we have our reference documents, we need to prepare them for search.
Because GPT can only read a limited amount of text at once, we'll split each document into chunks short enough to be read.
For this specific example on Wikipedia articles, we'll:
- Discard less relevant-looking sections like External Links and Footnotes
- Clean up the text by removing reference tags (e.g., ), whitespace, and super short sections
- Split each article into sections
- Prepend titles and subtitles to each section's text, to help GPT understand the context
- If a section is long (say, > 1,600 tokens), we'll recursively split it into smaller sections, trying to split along semantic boundaries like paragraphs
Next, we'll recursively split long sections into smaller sections.
There's no perfect recipe for splitting text into sections.
Some tradeoffs include:
- Longer sections may be better for questions that require more context
- Longer sections may be worse for retrieval, as they may have more topics muddled together
- Shorter sections are better for reducing costs (which are proportional to the number of tokens)
- Shorter sections allow more sections to be retrieved, which may help with recall
- Overlapping sections may help prevent answers from being cut by section boundaries
Here, we'll use a simple approach and limit sections to 1,600 tokens each, recursively halving any sections that are too long. To avoid cutting in the middle of useful sentences, we'll split along paragraph boundaries when possible.
3. Embed document chunks
Now that we've split our library into shorter self-contained strings, we can compute embeddings for each.
(For large embedding jobs, use a script like api_request_parallel_processor.py to parallelize requests while throttling to stay under rate limits.)
4. Store document chunks and embeddings
Because this example only uses a few thousand strings, we'll store them in a CSV file.
(For larger datasets, use a vector database, which will be more performant.)
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.