Load Stripe Data with Airbyte
LlamaIndex reader that loads Stripe objects into documents via Airbyte's Stripe connector.
Why it matters
Effortlessly extract and load various Stripe objects into your data pipelines. This asset leverages Airbyte's robust connector to pull data, enabling seamless integration with your existing systems.
Outcomes
What it gets done
Connect to Stripe using Airbyte's integration.
Extract specific Stripe data streams like invoices.
Load data incrementally, only fetching updated records.
Configure data retrieval with custom parameters and date ranges.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-airbyte-stripe | bash Overview
Airbyte Stripe Loader
The Airbyte Stripe Loader pulls Stripe objects into LlamaIndex documents via Airbyte's Stripe source connector, with support for a custom record_handler, memory-efficient lazy loading, and incremental syncs that only fetch changed records. Use it when you need Stripe data loaded into a LlamaIndex pipeline, especially at scale or on a recurring sync. It requires an Airbyte Stripe source config: client secret, account ID, start date.
What it does
The Airbyte Stripe Loader lets you access different Stripe objects and load them as documents into LlamaIndex, using Airbyte's Stripe source connector under the hood. By default, all fields from a Stripe record are stored as document metadata and the document text is set to the JSON representation of the whole record.
When to use - and when NOT to
Use it when you need to pull Stripe data (like invoices) into a LlamaIndex pipeline, especially at scale -- it supports lazy loading for large result sets and incremental loads that only fetch records created or updated since the last run. It requires a Stripe/Airbyte connection config (client secret, account ID, start date), so it is not usable without first configuring the underlying Airbyte Stripe source.
Inputs and outputs
Install with:
pip install llama-index-readers-airbyte-stripe
Configure and load data by stream name:
from llama_index.readers.airbyte_stripe import AirbyteStripeReader
stripe_config = {
# ...
}
reader = AirbyteStripeReader(config=stripe_config)
documents = reader.load_data(stream_name="invoices")
The config object follows Airbyte's Stripe source JSON schema, generally shaped as:
{
"client_secret": "<secret key>",
"account_id": "<account id>",
"start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",
}
To control how each Stripe record becomes a Document, pass a record_handler function. For large streams, reader.lazy_load_data returns an iterator instead of collecting everything into memory at once. For incremental syncs, reader.last_state can be saved and passed back into a later load_data(stream_name=..., state=current_state) call to fetch only what changed since last time.
Who it's for
Developers building LlamaIndex pipelines that need Stripe data -- invoices and other Stripe objects -- loaded efficiently and kept in sync incrementally, without writing their own Stripe API integration.
Source README
Airbyte Stripe Loader
pip install llama-index-readers-airbyte-stripe
The Airbyte Stripe Loader allows you to access different Stripe objects.
Usage
Here's an example usage of the AirbyteStripeReader.
from llama_index.readers.airbyte_stripe import AirbyteStripeReader
stripe_config = {
# ...
}
reader = AirbyteStripeReader(config=stripe_config)
documents = reader.load_data(stream_name="invoices")
Configuration
Check out the Airbyte documentation page for details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-stripe/source_stripe/spec.yaml.
The general shape looks like this:
{
"client_secret": "<secret key>",
"account_id": "<account id>",
"start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",
}
By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing a record_handler to the reader:
def handle_record(record, id):
return Document(
doc_id=id, text=record.data["title"], extra_info=record.data
)
reader = AirbyteStripeReader(
config=stripe_config, record_handler=handle_record
)
Lazy loads
The reader.load_data endpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By using reader.lazy_load_data instead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.
Incremental loads
This loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):
reader = AirbyteStripeReader(config={...})
documents = reader.load_data(stream_name="invoices")
current_state = reader.last_state # can be pickled away or stored otherwise
updated_documents = reader.load_data(
stream_name="invoices", state=current_state
) # only loads documents that were updated since last time
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.