Load Salesforce Data into LlamaIndex
A LlamaIndex data loader that connects to Salesforce via Airbyte to retrieve Salesforce objects for indexing applications.
Why it matters
Seamlessly extract and load data from various Salesforce objects into your LlamaIndex ecosystem. This asset facilitates efficient data retrieval for further processing and analysis.
Outcomes
What it gets done
Connect to Salesforce using OAuth credentials.
Specify which Salesforce objects (streams) to load.
Handle data extraction with customizable record handling.
Support incremental data loading to capture changes.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-airbyte-salesforce | bash Overview
Airbyte Salesforce Loader
The Airbyte Salesforce Loader retrieves Salesforce objects through the Airbyte connector and converts them into LlamaIndex documents. It supports filtering by object name patterns, custom record handling to control document text and metadata, and lazy loading for memory-efficient processing of large datasets. The loader tracks sync state to enable incremental updates that only fetch changed records. Use this loader when building LlamaIndex applications that need access to Salesforce data. It's particularly valuable when you need incremental syncing to avoid reprocessing unchanged records or when you want to filter specific Salesforce objects using pattern-based criteria.
What it does
The Airbyte Salesforce Loader is a LlamaIndex integration that retrieves data from Salesforce objects through the Airbyte connector framework. It loads Salesforce records as LlamaIndex documents, enabling you to index and query your Salesforce data within LlamaIndex-powered applications.
When to use - and when NOT to
Use this loader when you need to bring Salesforce data into a LlamaIndex pipeline. It supports filtering specific Salesforce objects using criteria patterns and provides incremental updates to avoid reprocessing unchanged records.
Consider your specific requirements when evaluating this loader against other integration approaches for your use case.
Inputs and outputs
You provide an OAuth configuration object containing your Salesforce credentials (client ID, client secret, refresh token), a start date for record retrieval in ISO format, sandbox environment flag, and streams criteria that define which Salesforce objects to load using filters like "exacts", "starts with", "contains", or their negations.
You receive LlamaIndex Document objects. By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. You can customize document construction by passing a record_handler function.
pip install llama-index-readers-airbyte-salesforce
from llama_index.readers.airbyte_salesforce import AirbyteSalesforceReader
salesforce_config = {
"client_id": "<oauth client id>",
"client_secret": "<oauth client secret>",
"refresh_token": "<oauth refresh token>",
"start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",
"is_sandbox": False,
"streams_criteria": [
{
"criteria": "exacts",
"value": "Account",
},
{"criteria": "starts with", "value": "Asset"}
]
}
reader = AirbyteSalesforceReader(config=salesforce_config)
documents = reader.load_data(stream_name="asset")
Integrations
Salesforce: Connects to Salesforce via OAuth to retrieve objects. Configuration follows the Airbyte Salesforce source connector specification.
LlamaIndex: Designed as a native data loader for the LlamaIndex framework, producing Document objects compatible with LlamaIndex indexing and querying pipelines.
Airbyte: Built on the Airbyte connector framework, following Airbyte's configuration schema and connection patterns for Salesforce integration.
Who it's for
Developers building LlamaIndex applications who need to load Salesforce data. Teams implementing incremental data pipelines where only updated Salesforce records need reprocessing, using the loader's state management to track the last sync point and load only changed documents.
Source README
Airbyte Salesforce Loader
pip install llama-index-readers-airbyte-salesforce
The Airbyte Salesforce Loader allows you to access different Salesforce objects.
Usage
Here's an example usage of the AirbyteSalesforceReader.
from llama_index.readers.airbyte_salesforce import AirbyteSalesforceReader
salesforce_config = {
# ...
}
reader = AirbyteSalesforceReader(config=salesforce_config)
documents = reader.load_data(stream_name="asset")
Configuration
Check out the Airbyte documentation page for details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce/source_salesforce/spec.yaml.
The general shape looks like this:
{
"client_id": "<oauth client id>",
"client_secret": "<oauth client secret>",
"refresh_token": "<oauth refresh token>",
"start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",
"is_sandbox": False, # set to True if you're using a sandbox environment
"streams_criteria": [ # Array of filters for salesforce objects that should be loadable
{
"criteria": "exacts",
"value": "Account",
}, # Exact name of salesforce object
{"criteria": "starts with", "value": "Asset"}, # Prefix of the name
# Other allowed criteria: ends with, contains, starts not with, ends not with, not contains, not exacts
],
}
By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing a record_handler to the reader:
def handle_record(record, id):
return Document(
doc_id=id, text=record.data["title"], extra_info=record.data
)
reader = AirbyteSalesforceReader(
config=salesforce_config, record_handler=handle_record
)
Lazy loads
The reader.load_data endpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By using reader.lazy_load_data instead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.
Incremental loads
This loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):
reader = AirbyteSalesforceReader(config={...})
documents = reader.load_data(stream_name="asset")
current_state = reader.last_state # can be pickled away or stored otherwise
updated_documents = reader.load_data(
stream_name="asset", state=current_state
) # only loads documents that were updated since last time
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.