Fetch and Process Scientific Papers
LlamaIndex loaders that pull scientific papers from Arxiv or PubMed by search query.
Why it matters
Access and process scientific literature from Arxiv and Pubmed. This asset extracts abstracts and full text from relevant papers based on your search queries, making them ready for further analysis or integration into your knowledge base.
Outcomes
What it gets done
Retrieve scientific papers from Arxiv based on search queries.
Extract abstracts and full text from fetched papers.
Load paper data into LlamaIndex for further processing.
Fetch scientific papers from Pubmed based on search queries.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-papers | bash Overview
Papers Loaders
The Papers Loaders fetch scientific literature into LlamaIndex documents from two sources: ArxivReader for Arxiv papers (abstracts split into separate documents) and PubmedReader for PubMed papers, both searchable by query. Use it when you need Arxiv or PubMed papers loaded into LlamaIndex by search query. Both default to 10 papers per query, configurable higher or lower.
What it does
The Papers Loaders fetch text from the most relevant scientific papers matching a search query, via two readers: ArxivReader (fetches papers from Arxiv, extracting each abstract into a separate document) and PubmedReader (fetches papers from PubMed, including each abstract in the returned Document). Arxiv's search query can be any string, an Arxiv paper ID, or a full Arxiv query string using Arxiv's own query syntax (field prefixes like au: for author, documented in Arxiv's API user manual); PubMed's search query can be any string.
When to use - and when NOT to
Use it when you need scientific literature -- from Arxiv or PubMed -- pulled into LlamaIndex documents by search query, for example "au:Karpathy" on Arxiv or "amyloidosis" on PubMed. Both readers default to parsing up to 10 papers per query (configurable), and the Arxiv reader optionally lets you specify a local directory to temporarily store paper PDFs, which are deleted automatically afterward.
Inputs and outputs
Install with:
pip install llama-index-readers-papers
For Arxiv:
from llama_index.readers.papers import ArxivReader
loader = ArxivReader()
documents = loader.load_data(search_query="au:Karpathy")
To load papers and abstracts as separate return values instead of combined documents, use load_papers_and_abstracts(search_query=...) instead of load_data.
For PubMed:
from llama_index.readers.papers import PubmedReader
loader = PubmedReader()
documents = loader.load_data(search_query="amyloidosis")
Who it's for
Developers building LlamaIndex pipelines that need scientific literature from Arxiv or PubMed loaded and searchable by query, with abstracts available either combined into the document or returned separately.
Source README
Papers Loaders
pip install llama-index-readers-papers
Arxiv Papers Loader
This loader fetches the text from the most relevant scientific papers on Arxiv specified by a search query (e.g. "Artificial Intelligence"). For each paper, the abstract is extracted and put in a separate document. The search query may be any string, Arxiv paper id, or a general Arxiv query string (see the full list of capabilities here).
Usage
To use this loader, you need to pass in the search query. You may also optionally specify a local directory to temporarily store the paper PDFs (they are deleted automatically) and the maximum number of papers you want to parse for your search query (default is 10).
from llama_index.readers.papers import ArxivReader
loader = ArxivReader()
documents = loader.load_data(search_query="au:Karpathy")
Alternatively, if you would like to load papers and abstracts separately:
from llama_index.readers.papers import ArxivReader
loader = ArxivReader()
documents, abstracts = loader.load_papers_and_abstracts(
search_query="au:Karpathy"
)
This loader is designed to be used as a way to load data into LlamaIndex.
Pubmed Papers Loader
This loader fetches the text from the most relevant scientific papers on Pubmed specified by a search query (e.g. "Alzheimers"). For each paper, the abstract is included in the Document. The search query may be any string.
Usage
To use this loader, you need to pass in the search query. You may also optionally specify the maximum number of papers you want to parse for your search query (default is 10).
from llama_index.readers.papers import PubmedReader
loader = PubmedReader()
documents = loader.load_data(search_query="amyloidosis")
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.