Extract Wikipedia content into LlamaIndex documents
Load the text content of specific Wikipedia pages into LlamaIndex documents.
Why it matters
Retrieve and load text content from Wikipedia pages into LlamaIndex for indexing, search, and retrieval-augmented generation workflows.
Outcomes
What it gets done
Specify a list of Wikipedia page titles to retrieve
Extract full text content from each Wikipedia page
Convert Wikipedia content into LlamaIndex document format
Enable Wikipedia data as a tool for LangChain agents
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-wikipedia | bash Overview
LlamaIndex Readers Integration: Wikipedia
A LlamaIndex reader that loads the text content of specific, named Wikipedia pages into documents. Use when you already know which Wikipedia pages you need indexed, not for general Wikipedia search.
What it does
The Wikipedia Reader reads Wikipedia pages and retrieves their content, letting a developer specify a list of pages to read and pulling the text content of each one into LlamaIndex documents. WikipediaReader is initialized with no configuration, and load_data takes a pages list of page titles to load.
When to use - and when NOT to
Use it when you need the text content of specific, named Wikipedia pages loaded into LlamaIndex - for example building a reference index over a known set of topics. Do not use it as a general web search or discovery tool; it only loads pages you name explicitly by title, it does not search Wikipedia for relevant pages on your behalf.
Capabilities
load_data takes a list of Wikipedia page titles and returns the text content of each as a LlamaIndex document.
How to install
from llama_index.readers.wikipedia import WikipediaReader
reader = WikipediaReader()
documents = reader.load_data(pages=["Page Title 1", "Page Title 2", ...])
Who it's for
Developers who need the content of specific, known Wikipedia pages loaded into LlamaIndex for indexing and retrieval.
Source README
LlamaIndex Readers Integration: Wikipedia
Overview
The Wikipedia Reader reads Wikipedia pages and retrieves their content. It allows you to specify a list of pages to read, and it retrieves the text content of each page.
Installation
You can install the Wikipedia Reader via pip:
pip install llama-index-readers-wikipedia
Usage
from llama_index.readers.wikipedia import WikipediaReader
### Initialize WikipediaReader
reader = WikipediaReader()
### Load data from Wikipedia
documents = reader.load_data(pages=["Page Title 1", "Page Title 2", ...])
This loader is designed to be used as a way to load data into
LlamaIndex and/or subsequently
used as a Tool in a LangChain Agent.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.