Load and Process UniProt Data with LlamaIndex
Parse UniProt Swiss-Prot protein data files into LlamaIndex documents, with lazy loading for multi-GB files.
Why it matters
Integrate UniProt protein data into your LlamaIndex applications. This reader efficiently parses large UniProt files, enabling structured access to protein information for advanced analysis and knowledge base construction.
Outcomes
What it gets done
Parse UniProt Swiss-Prot format files.
Load protein data into LlamaIndex structures.
Enable lazy loading for memory-efficient processing of large datasets.
Selectively include specific protein fields for customized data extraction.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-uniprot | bash Overview
UniProt Reader for LlamaIndex
A LlamaIndex reader that parses UniProt Swiss-Prot protein data files into documents, with lazy loading for large files and configurable field selection. Use it to index UniProt protein data in LlamaIndex. Prefer lazy_load_data over load_data for multi-gigabyte Swiss-Prot files to avoid loading everything into memory.
What it does
llama-index-readers-uniprot parses UniProt Swiss-Prot format protein-database files, loading individual protein records into LlamaIndex as documents. Each document's text holds the entire UniProt record, and its metadata carries the protein ID, giving a structured output split cleanly between full-record text and queryable metadata, with configurable field selection to control what's included.
When to use - and when NOT to
Use this reader when you need to index UniProt protein data for search or retrieval in LlamaIndex, especially since Swiss-Prot files typically run several gigabytes. For files at that scale, use lazy_load_data rather than load_data so records are processed one at a time instead of loading the entire database into memory at once - load_data alone is only practical for small files or when you've already limited the record count with max_records.
Inputs and outputs
load_data(path) returns a list of Document objects with an id in metadata; lazy_load_data(path) yields them one at a time instead:
reader = UniProtReader()
documents = reader.load_data("path/to/uniprot_sprot.dat")
for doc in documents:
print(f"Protein ID: {doc.metadata['id']}")
Both methods accept max_records to cap how many entries get parsed, useful for testing or incremental loads - for example, parsing only the first 1,000 records with UniProtReader(max_records=1000), which works with lazy loading too. Available fields include id, accession, description, gene_names, organism, comments, keywords, sequence_length, sequence_mw, taxonomy, taxonomy_id, citations, cross_references, and features - covering everything from basic identifiers and gene names to sequence length and molecular weight, taxonomic classification, literature citations, and cross-references to other databases. All fields are included by default, or narrowed with include_fields, for example {"id", "description", "sequence"} to keep only the essentials.
Integrations
For incremental indexing over a large file, the project's own example pairs lazy_load_data with a running set of already-indexed protein IDs (read from the index's docstore), skipping documents already present and calling index.refresh_ref_docs() in fixed-size batches (for example, 10 documents at a time) as new ones accumulate, using a SentenceSplitter to chunk text before persisting the index with storage_context.persist(). Install with pip install llama-index-readers-uniprot.
Who it's for
Bioinformatics and life-sciences teams building LlamaIndex search or retrieval over UniProt protein data, especially over the full multi-gigabyte Swiss-Prot database where memory-efficient, incremental loading matters. The reader is built specifically for efficient parsing of large UniProt files, rather than as a general-purpose text loader repurposed for this format, and the project accepts external contributions per its standard contributing guidelines in the main LlamaIndex repository.
Source README
UniProt Reader for LlamaIndex
This package provides a reader for UniProt Swiss-Prot format files, allowing you to load protein data into LlamaIndex for further processing and analysis.
Features
- Efficient parsing of large UniProt files with optional lazy loading.
- Structured output with both text containing entire UniProt record and metadata containing protein ID.
- Configurable field selection
Installation
pip install llama-index-readers-uniprot
Usage
from llama_index.readers.uniprot import UniProtReader
### Initialize the reader
reader = UniProtReader()
### Load data from a UniProt file
documents = reader.load_data("path/to/uniprot_sprot.dat")
### Access the documents
for doc in documents:
print(f"Protein ID: {doc.metadata['id']}")
Lazy Loading for Large Files
Since UniProt files are large (several GB) it's recommended to use lazy loading to process records one at a time,
without loading the entire database into memory:
### Initialize the reader
reader = UniProtReader()
### Load data lazily from a UniProt file
for doc in reader.lazy_load_data("path/to/uniprot_sprot.dat"):
print(f"Protein ID: {doc.metadata['id']}")
print("---")
Example of building an index from a lazy loaded UniProt file
from llama_index.readers.uniprot import UniProtReader
from llama_index.core import VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter
reader = UniProtReader(max_records=10000)
### Load existing protein IDs from the index
existing_protein_ids = {
node.metadata.get('id')
for node in index.storage_context.docstore.docs.values()
if node.metadata.get('id')
}
text_splitter = SentenceSplitter(chunk_size=2048)
index = VectorStoreIndex([], transformations=[text_splitter], show_progress=True)
documents_gen = reader.lazy_load_data("path/to/uniprot_sprot.dat")
### Process documents in batches
batch_size = 10
current_batch = []
for doc in documents_gen:
protein_id = doc.metadata.get('id')
if protein_id in existing_protein_ids:
print(f"Skipping document {protein_id} - already indexed")
continue
current_batch.append(doc)
if len(current_batch) >= batch_size:
index.refresh_ref_docs(documents=current_batch)
current_batch = []
### Process any remaining documents
if current_batch:
index.refresh_ref_docs(documents=current_batch)
### Define persist directory
persist_dir = "path/to/persist/directory"
index.storage_context.persist(persist_dir=persist_dir)
Customizing Field Selection
You can specify which fields to include in the output:
### Only include specific fields
reader = UniProtReader(include_fields={"id", "description", "sequence"})
documents = reader.load_data("path/to/uniprot_sprot.dat")
Available fields:
id: Protein identifieraccession: Accession numbersdescription: Protein descriptiongene_names: Gene namesorganism: Organism namecomments: Comments and annotationskeywords: Keywordssequence_length: Length of the protein sequencesequence_mw: Molecular weight of the proteintaxonomy: Taxonomic classificationtaxonomy_id: Taxonomic database identifierscitations: Literature citationscross_references: Cross-references to other databasesfeatures: Protein features
By default, all fields are included.
Limiting Number of Records
You can limit the number of records to parse using the max_records parameter:
### Parse only first 1000 records
reader = UniProtReader(max_records=1000)
documents = reader.load_data("path/to/uniprot_sprot.dat")
### Works with lazy loading too
for doc in reader.lazy_load_data(
"path/to/uniprot_sprot.dat", max_records=1000
):
print(f"Protein ID: {doc.metadata['id']}")
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.