Load Typeform Data with Airbyte
Load Typeform objects into LlamaIndex via the Airbyte Typeform connector, with incremental sync.
Why it matters
Effortlessly ingest data from Typeform into your AI applications. This asset connects to Typeform via Airbyte, enabling seamless data extraction for further processing and analysis.
Outcomes
What it gets done
Connect to Typeform using Airbyte.
Extract form data and responses.
Load data incrementally or in full.
Prepare Typeform data for AI indexing.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-airbyte-typeform | bash Overview
Airbyte Typeform Loader
A LlamaIndex reader that loads Typeform data via Airbyte's Typeform connector, scoped by start date and form IDs, with lazy and incremental loading. Use for Typeform data needing date-bounded or form-scoped retrieval, with lazy loading for volume and incremental sync for recurring runs.
What it does
The Airbyte Typeform Loader gives access to different Typeform objects through the Airbyte Typeform connector, letting LlamaIndex load Typeform data such as forms without a custom integration. AirbyteTypeformReader is initialized with a config dictionary matching Airbyte's Typeform source spec, and load_data takes a stream_name (such as forms) identifying which Typeform object stream to pull.
The config object follows a documented JSON schema published in Airbyte's GitHub repository, and includes credentials (an auth_type of Private Token with an access_token), a start_date in ISO format bounding how far back records are retrieved, and an optional form_ids list - if omitted, records from all forms are loaded instead of being scoped to specific ones.
By default, every field of a record is stored as document metadata and the document text is the JSON representation of all fields; a custom record_handler function can be passed to construct the text and metadata differently, for example using just a record's title field as the document text. Beyond the standard load_data, lazy_load_data returns an iterator instead of a full list for streaming large record sets without holding everything in memory, and incremental loading lets the reader track state (reader.last_state, picklable for storage) so a later load_data call with that state returns only records that are new or updated since the last run.
When to use - and when NOT to
Use it when you need Typeform data - forms and their responses - loaded into LlamaIndex, scoped by a start date and optionally by specific form IDs. Use lazy_load_data for large response volumes to avoid memory pressure, and incremental loading for recurring syncs where only new or changed records matter. Do not omit start_date; the config schema requires it to bound the retrieval window.
Capabilities
load_data/lazy_load_data pull a named Typeform stream (such as forms) scoped by start date and optional form IDs, with default JSON-as-text document construction, an optional custom record_handler, and incremental loading via a trackable last_state.
How to install
pip install llama-index-readers-airbyte-typeform
Requires a Typeform config object with a private-token access_token, a start_date, and optionally specific form_ids.
Who it's for
Developers who need Typeform form and response data loaded into LlamaIndex, especially for large volumes (via lazy loading) or recurring incremental syncs.
Source README
Airbyte Typeform Loader
pip install llama-index-readers-airbyte-typeform
The Airbyte Typeform Loader allows you to access different Typeform objects.
Usage
Here's an example usage of the AirbyteTypeformReader.
from llama_index.readers.airbyte_typeform import AirbyteTypeformReader
typeform_config = {
# ...
}
reader = AirbyteTypeformReader(config=typeform_config)
documents = reader.load_data(stream_name="forms")
Configuration
Check out the Airbyte documentation page for details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-typeform/source_typeform/spec.json.
The general shape looks like this:
{
"credentials": {
"auth_type": "Private Token",
"access_token": "<your auth token>",
},
"start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",
"form_ids": [
"<id of form to load records for>"
], # if omitted, records from all forms will be loaded
}
By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing a record_handler to the reader:
def handle_record(record, id):
return Document(
doc_id=id, text=record.data["title"], extra_info=record.data
)
reader = AirbyteTypeformReader(
config=typeform_config, record_handler=handle_record
)
Lazy loads
The reader.load_data endpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By using reader.lazy_load_data instead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.
Incremental loads
This loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):
reader = AirbyteTypeformReader(config={...})
documents = reader.load_data(stream_name="forms")
current_state = reader.last_state # can be pickled away or stored otherwise
updated_documents = reader.load_data(
stream_name="forms", state=current_state
) # only loads documents that were updated since last time
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.