Load Stripe Data with Airbyte
Airbyte Stripe Loader syncs Stripe objects like invoices into LlamaIndex, with lazy and incremental loading.
Why it matters
Effortlessly extract and load various Stripe objects into your data pipelines. This asset leverages Airbyte's robust connector to pull data, enabling seamless integration with your existing systems.
Outcomes
What it gets done
Connect to Stripe using Airbyte's integration.
Extract specific Stripe data streams like invoices.
Load data incrementally, only fetching updated records.
Configure data retrieval with custom parameters and date ranges.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-airbyte-stripe | bash Overview
Airbyte Stripe Loader
Airbyte Stripe Loader loads Stripe payment objects like invoices into LlamaIndex documents via Airbyte's Stripe connector, supporting custom record formatting, lazy loading, and incremental sync by saved state. Use it when Stripe payment data needs to feed a LlamaIndex pipeline; requires a valid Stripe secret key and account ID.
What it does
Airbyte Stripe Loader gives access to different Stripe objects (like invoices) as LlamaIndex documents, built on Airbyte's Stripe source connector. It authenticates with a Stripe secret key and account ID, and by default stores every field as document metadata with the text set to the JSON representation of the record - customizable via a record_handler to shape the document text yourself.
When to use - and when NOT to
Use this when you want Stripe payment data - invoices or other objects Airbyte's Stripe connector supports - loaded directly into a LlamaIndex pipeline for financial reporting, search, or RAG. It supports lazy loading (lazy_load_data) for large transaction histories that shouldn't be held entirely in memory, and incremental loading so subsequent syncs only pull new or updated records using a saved state. It is not usable without a valid Stripe secret key and account ID, and available objects/fields depend on what Airbyte's Stripe connector exposes.
Inputs and outputs
Input: a Stripe config dict per Airbyte's spec (client_secret, account_id, start_date), a stream_name (e.g. "invoices"), and optionally a record_handler and/or saved state for incremental loads. Output: load_data returns a list of documents (or lazy_load_data returns an iterator); reader.last_state can be saved and passed back to fetch only what changed since the last load.
Integrations
Wraps Airbyte's Stripe source connector as a LlamaIndex AirbyteStripeReader, requiring llama-index-readers-airbyte-stripe and a Stripe secret key configured per Airbyte's documentation.
Who it's for
Teams using Stripe for payments who want invoice and transaction data loaded directly into a LlamaIndex pipeline, especially when incremental or memory-efficient loading matters for large transaction volumes.
Source README
Airbyte Stripe Loader
pip install llama-index-readers-airbyte-stripe
The Airbyte Stripe Loader allows you to access different Stripe objects.
Usage
Here's an example usage of the AirbyteStripeReader.
from llama_index.readers.airbyte_stripe import AirbyteStripeReader
stripe_config = {
# ...
}
reader = AirbyteStripeReader(config=stripe_config)
documents = reader.load_data(stream_name="invoices")
Configuration
Check out the Airbyte documentation page for details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/spec.yaml.
The general shape looks like this:
{
"client_secret": "<secret key>",
"account_id": "<account id>",
"start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",
}
By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing a record_handler to the reader:
def handle_record(record, id):
return Document(
doc_id=id, text=record.data["title"], extra_info=record.data
)
reader = AirbyteStripeReader(
config=stripe_config, record_handler=handle_record
)
Lazy loads
The reader.load_data endpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By using reader.lazy_load_data instead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.
Incremental loads
This loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):
reader = AirbyteStripeReader(config={...})
documents = reader.load_data(stream_name="invoices")
current_state = reader.last_state # can be pickled away or stored otherwise
updated_documents = reader.load_data(
stream_name="invoices", state=current_state
) # only loads documents that were updated since last time
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.