Load and Process UniProt Data with LlamaIndex
Parse UniProt Swiss-Prot protein files into LlamaIndex documents, with lazy loading for large files.
Why it matters
Integrate UniProt protein data into your LlamaIndex applications. This reader efficiently parses large UniProt files, enabling structured access to protein information for advanced analysis and knowledge base construction.
Outcomes
What it gets done
Parse UniProt Swiss-Prot format files.
Load protein data into LlamaIndex structures.
Enable lazy loading for memory-efficient processing of large datasets.
Selectively include specific protein fields for customized data extraction.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-uniprot | bash Overview
UniProt Reader for LlamaIndex
A LlamaIndex reader that parses UniProt Swiss-Prot protein files into documents, with lazy loading and field selection for large files. Use lazy_load_data for large multi-gigabyte files, include_fields to scope output, and max_records to test on a subset first.
What it does
The UniProt Reader parses UniProt Swiss-Prot format files, loading protein data into LlamaIndex for further processing and analysis. Each document produced contains the full text of a UniProt record along with metadata carrying the protein ID, and the reader supports efficient parsing of large files with optional lazy loading and configurable field selection.
UniProtReader is instantiated with no required arguments, and load_data takes the path to a .dat UniProt file, returning documents where each document's metadata includes an id field. Because UniProt files can run to several gigabytes, lazy_load_data is recommended for production use - it yields documents one at a time rather than loading the entire database into memory, and the source's own example shows this paired with batched index updates (accumulating documents into a batch, then calling index.refresh_ref_docs) to build a VectorStoreIndex incrementally while skipping protein IDs already indexed. That same worked example configures a SentenceSplitter with a 2048-token chunk size for the index and persists the finished index to a specified directory afterward via storage_context.persist.
Field selection is configurable via include_fields, restricting output to just the named fields (for example {"id", "description", "sequence"}) instead of every available field; by default, all fields are included. The full set of available fields covers id, accession, description, gene_names, organism, comments, keywords, sequence_length, sequence_mw, taxonomy, taxonomy_id, citations, cross_references, and features. A max_records parameter, usable with either load_data or lazy_load_data, caps how many records get parsed - useful for testing against a subset before running against a full multi-gigabyte file.
When to use - and when NOT to
Use it when you need protein data from a UniProt Swiss-Prot .dat file loaded into LlamaIndex for retrieval or analysis, using lazy_load_data for any file large enough that loading it all into memory at once would be impractical. Use include_fields to scope output to just the fields you actually need, and max_records to test against a small subset before a full run. Do not use load_data (non-lazy) on multi-gigabyte production files without first confirming memory can accommodate the full parse.
Capabilities
load_data/lazy_load_data parse a UniProt .dat file into documents (full record text plus protein-ID metadata), with configurable include_fields scoping and a max_records cap for testing or partial parsing.
How to install
pip install llama-index-readers-uniprot
Who it's for
Developers and researchers who need protein data from UniProt Swiss-Prot files loaded into LlamaIndex for analysis or retrieval, especially at the scale of large, multi-gigabyte database files.
Source README
UniProt Reader for LlamaIndex
This package provides a reader for UniProt Swiss-Prot format files, allowing you to load protein data into LlamaIndex for further processing and analysis.
Features
- Efficient parsing of large UniProt files with optional lazy loading.
- Structured output with both text containing entire UniProt record and metadata containing protein ID.
- Configurable field selection
Installation
pip install llama-index-readers-uniprot
Usage
from llama_index.readers.uniprot import UniProtReader
### Initialize the reader
reader = UniProtReader()
### Load data from a UniProt file
documents = reader.load_data("path/to/uniprot_sprot.dat")
### Access the documents
for doc in documents:
print(f"Protein ID: {doc.metadata['id']}")
Lazy Loading for Large Files
Since UniProt files are large (several GB) it's recommended to use lazy loading to process records one at a time,
without loading the entire database into memory:
### Initialize the reader
reader = UniProtReader()
### Load data lazily from a UniProt file
for doc in reader.lazy_load_data("path/to/uniprot_sprot.dat"):
print(f"Protein ID: {doc.metadata['id']}")
print("---")
Example of building an index from a lazy loaded UniProt file
from llama_index.readers.uniprot import UniProtReader
from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter
reader = UniProtReader(max_records=10000)
### Load existing protein IDs from the index
existing_protein_ids = {
node.metadata.get('id')
for node in index.storage_context.docstore.docs.values()
if node.metadata.get('id')
}
text_splitter = SentenceSplitter(chunk_size=2048)
index = VectorStoreIndex([], transformations=[text_splitter], show_progress=True)
documents_gen = reader.lazy_load_data("path/to/uniprot_sprot.dat")
### Process documents in batches
batch_size = 10
current_batch = []
for doc in documents_gen:
protein_id = doc.metadata.get('id')
if protein_id in existing_protein_ids:
print(f"Skipping document {protein_id} - already indexed")
continue
current_batch.append(doc)
if len(current_batch) >= batch_size:
index.refresh_ref_docs(documents=current_batch)
current_batch = []
### Process any remaining documents
if current_batch:
index.refresh_ref_docs(documents=current_batch)
### Define persist directory
persist_dir = "path/to/persist/directory"
index.storage_context.persist(persist_dir=persist_dir)
Customizing Field Selection
You can specify which fields to include in the output:
### Only include specific fields
reader = UniProtReader(include_fields={"id", "description", "sequence"})
documents = reader.load_data("path/to/uniprot_sprot.dat")
Available fields:
id: Protein identifieraccession: Accession numbersdescription: Protein descriptiongene_names: Gene namesorganism: Organism namecomments: Comments and annotationskeywords: Keywordssequence_length: Length of the protein sequencesequence_mw: Molecular weight of the proteintaxonomy: Taxonomic classificationtaxonomy_id: Taxonomic database identifierscitations: Literature citationscross_references: Cross-references to other databasesfeatures: Protein features
By default, all fields are included.
Limiting Number of Records
You can limit the number of records to parse using the max_records parameter:
### Parse only first 1000 records
reader = UniProtReader(max_records=1000)
documents = reader.load_data("path/to/uniprot_sprot.dat")
### Works with lazy loading too
for doc in reader.lazy_load_data(
"path/to/uniprot_sprot.dat", max_records=1000
):
print(f"Protein ID: {doc.metadata['id']}")
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.