Extract Shopify Data with Airbyte
Airbyte Shopify Loader is a LlamaIndex reader that loads Shopify store data into documents for AI indexing and retrieval applications.
Why it matters
Seamlessly extract data from your Shopify store using the Airbyte connector. Load product information, order details, and customer data for analysis or integration into other systems.
Outcomes
What it gets done
Connect to your Shopify store via Airbyte.
Extract various Shopify objects like orders, products, and customers.
Support for incremental data loading to capture changes.
Configure data retrieval based on start date and shop name.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-airbyte-shopify | bash Overview
Airbyte Shopify Loader
The Airbyte Shopify Loader is a LlamaIndex reader that extracts data from Shopify stores and converts it into document objects. It accesses Shopify objects through configuration that references the Airbyte connector specification. The loader supports customizable document construction, lazy loading for large datasets, and incremental syncing to load only updated records. Use this loader when building LlamaIndex applications that need to incorporate Shopify store data. It's especially valuable when you need incremental data syncing to track only changes since the last load, or when processing large datasets that require memory-efficient lazy loading.
What it does
The Airbyte Shopify Loader is a LlamaIndex integration that connects to Shopify stores and loads Shopify objects as documents. It transforms Shopify data into LlamaIndex-compatible document formats. The source material demonstrates loading orders as an example stream.
When to use - and when NOT to
Use this loader when you need to incorporate Shopify store data into LlamaIndex applications. It's ideal for incremental data syncing scenarios where you need to track only updated records since the last load.
Do not use this loader if you're working with non-Shopify e-commerce platforms that require different connectors.
Inputs and outputs
You provide a configuration object containing your Shopify store credentials (shop name, API password, start date for data retrieval). The configuration follows this structure:
{
"start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",
"shop": "<name of the shop you want to retrieve documents from>",
"credentials": {
"auth_method": "api_password",
"api_password": "<your api password>",
},
}
You receive LlamaIndex Document objects with Shopify data stored as metadata and text fields. By default, all fields are stored as metadata and the text is set to the JSON representation of all fields. You can customize document construction by passing a record_handler function.
Integrations
Configuration details follow the Airbyte Shopify source specification.
Who it's for
This tool serves developers building LlamaIndex applications that need access to Shopify data.
The loader supports both standard batch loading and lazy loading for memory-efficient processing of large datasets. It also provides incremental loading capabilities, allowing you to load only documents that have been updated since the last sync:
reader = AirbyteShopifyReader(config={...})
documents = reader.load_data(stream_name="orders")
current_state = reader.last_state
updated_documents = reader.load_data(
stream_name="orders", state=current_state
)
Installation is straightforward via pip:
pip install llama-index-readers-airbyte-shopify
Source README
Airbyte Shopify Loader
pip install llama-index-readers-airbyte-shopify
The Airbyte Shopify Loader allows you to access different Shopify objects.
Usage
Here's an example usage of the AirbyteShopifyReader.
from llama_index.readers.airbyte_shopify import AirbyteShopifyReader
shopify_config = {
# ...
}
reader = AirbyteShopifyReader(config=shopify_config)
documents = reader.load_data(stream_name="orders")
Configuration
Check out the Airbyte documentation page for details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-shopify/source_shopify/spec.json.
The general shape looks like this:
{
"start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",
"shop": "<name of the shop you want to retrieve documents from>",
"credentials": {
"auth_method": "api_password",
"api_password": "<your api password>",
},
}
By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing a record_handler to the reader:
def handle_record(record, id):
return Document(
doc_id=id, text=record.data["title"], extra_info=record.data
)
reader = AirbyteShopifyReader(
config=shopify_config, record_handler=handle_record
)
Lazy loads
The reader.load_data endpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By using reader.lazy_load_data instead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.
Incremental loads
This loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):
reader = AirbyteShopifyReader(config={...})
documents = reader.load_data(stream_name="orders")
current_state = reader.last_state # can be pickled away or stored otherwise
updated_documents = reader.load_data(
stream_name="orders", state=current_state
) # only loads documents that were updated since last time
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.