Search and Load Open Access Papers from OpenAlex
Search OpenAlex for papers, including full-text search of Open Access works, into LlamaIndex.
Why it matters
Access and process scholarly literature from OpenAlex, enabling full-text searches for open access papers to support research and analysis.
Outcomes
What it gets done
Search OpenAlex for academic papers based on a query.
Load paper data into a structured format for further processing.
Enable full-text search capabilities for open access publications.
Retrieve answers with citations from the loaded paper data.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-openalex | bash Overview
OpenAlex Reader
A LlamaIndex reader that searches OpenAlex for papers, with optional full-text search of Open Access works. Use for citation-backed research queries, and full_text=True specifically for searching inside Open Access paper content.
What it does
The OpenAlex Reader searches for papers in OpenAlex and loads them into LlamaIndex. Its main advantage over similar loaders is that it can search the full text of Open Access papers, not just titles and abstracts, when that option is enabled.
OpenAlexReader is initialized with an email address, and load_data takes a query string plus a full_text boolean - set to True to search full-text content of Open Access papers, or left False for the default metadata-level search. The source's own worked example queries "biases in large language models" and, once indexed, a query engine call produces a markdown table of biases with citations - showing results like data selection bias, social bias across several categories, and anti-Muslim and anti-LGBTQ+ bias, each attributed to a specific numbered source.
When to use - and when NOT to
Use it when you need to search and retrieve academic papers - grounded with citations to specific sources - for a research topic, and specifically use full_text=True when abstract-level metadata isn't enough and you need to search inside the actual paper content of Open Access works. Do not expect full-text search to work for papers that are not Open Access; that capability is scoped to openly accessible content.
Capabilities
load_data searches OpenAlex for papers matching a query, optionally searching full text for Open Access papers, and returns them as LlamaIndex documents ready for citation-backed querying.
How to install
pip install llama-index-readers-openalex
Requires an email address passed to OpenAlexReader (OpenAlex's API convention for identifying callers).
Who it's for
Researchers and developers building LlamaIndex applications that need citation-backed answers grounded in OpenAlex papers, including full-text search of Open Access works.
Source README
OpenAlex Reader
pip install llama-index-readers-openalex
This loader will search for papers in OpenAlex and load them in llama-index. The main advantage of using OpenAlex is that you can search the full-text for Open Access papers as well.
Usage
from llama_index.readers.openalex import OpenAlexReader
openalex_reader = OpenAlexReader(email="shauryr@gmail.com")
query = "biases in large language models"
### changing this to full_text=True will let you search full-text
documents = openalex_reader.load_data(query, full_text=False)
What can it do?
As shown in demo.ipynb we can get answers with citations.
query = "biases in large language models"
response = query_engine.query(
"list the biases in large language models in a markdown table"
)
Output
| Source | Biases |
|---|---|
| Source 1 | Data selection bias, social bias (gender, age, sexual orientation, ethnicity, religion, culture) |
| Source 2 | Biases of what is right and wrong to do, reflecting ethical and moral norms of society |
| Source 3 | Anti-Muslim bias |
| Source 6 | Gender bias |
| Source 9 | Anti-LGBTQ+ bias |
| Source 10 | Potential bias in the output |
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.