Sync HubSpot Data with Airbyte
Airbyte Hubspot Loader is a LlamaIndex reader that loads Hubspot objects into documents with support for incremental syncs and custom record handling.
Why it matters
Effortlessly extract and synchronize data from HubSpot into your data pipelines. This asset enables seamless integration with Airbyte, allowing you to leverage your HubSpot information for analysis and other downstream applications.
Outcomes
What it gets done
Connect to HubSpot using Airbyte.
Extract various HubSpot objects (e.g., products).
Handle data incrementally and lazily for efficiency.
Configure data retrieval with custom record handlers.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-airbyte-hubspot | bash Overview
Airbyte Hubspot Loader
The Airbyte Hubspot Loader is a LlamaIndex reader that connects to Hubspot via Airbyte and loads CRM objects into document format. It converts Hubspot records into LlamaIndex documents with all fields stored as metadata and supports custom record handlers to control document construction. The loader offers both standard and lazy loading modes, plus incremental sync to fetch only updated records. Use this loader when building LLM applications or search indexes over Hubspot CRM data, especially when you need to keep document stores synchronized with Hubspot through incremental updates. It's designed for use within the LlamaIndex framework.
What it does
The Airbyte Hubspot Loader is a data reader for LlamaIndex that connects to Hubspot via Airbyte and loads Hubspot objects into document format. It accesses different Hubspot objects and converts them into LlamaIndex documents, with all fields stored as metadata and the full JSON representation as text by default.
When to use - and when NOT to
Use this loader when you need to build LLM applications or search indexes over your Hubspot CRM data. It's particularly valuable when you need incremental updates to keep your document store synchronized with Hubspot changes without reloading everything. Do not use this if you need real-time streaming data - this is a batch loader that pulls snapshots at the time of execution. Avoid this tool if you're working outside the LlamaIndex ecosystem, as it's specifically designed to produce LlamaIndex Document objects.
Inputs and outputs
You provide a configuration object containing your Hubspot credentials (Private App access token), a start date in ISO format, and the name of the stream you want to load (e.g., "products"). Optionally, you can provide a custom record handler function to control how records are transformed into documents, and a state object for incremental loading.
You receive documents as a list or iterator. By default, all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. When performing incremental loads, you also receive a state object that can be persisted and reused to fetch only updated records on subsequent runs.
Integrations
This loader integrates with Hubspot via Airbyte and is designed specifically for LlamaIndex.
Who it's for
This tool is for developers building LLM-powered applications with LlamaIndex who need to incorporate Hubspot CRM data.
Installation and basic usage
pip install llama-index-readers-airbyte-hubspot
from llama_index.readers.airbyte_hubspot import AirbyteHubspotReader
hubspot_config = {
"start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",
"credentials": {
"credentials_title": "Private App Credentials",
"access_token": "<access token of your private app>",
},
}
reader = AirbyteHubspotReader(config=hubspot_config)
documents = reader.load_data(stream_name="products")
Incremental loading example
reader = AirbyteHubspotReader(config={...})
documents = reader.load_data(stream_name="products")
current_state = reader.last_state
updated_documents = reader.load_data(
stream_name="products", state=current_state
)
Source README
Airbyte Hubspot Loader
pip install llama-index-readers-airbyte-hubspot
The Airbyte Hubspot Loader allows you to access different Hubspot objects.
Usage
Here's an example usage of the AirbyteHubspotReader.
from llama_index.readers.airbyte_hubspot import AirbyteHubspotReader
hubspot_config = {
# ...
}
reader = AirbyteHubspotReader(config=hubspot_config)
documents = reader.load_data(stream_name="products")
Configuration
Check out the Airbyte documentation page for details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-hubspot/source_hubspot/spec.yaml.
The general shape looks like this:
{
"start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",
"credentials": {
"credentials_title": "Private App Credentials",
"access_token": "<access token of your private app>",
},
}
By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing a record_handler to the reader:
def handle_record(record, id):
return Document(
doc_id=id, text=record.data["title"], extra_info=record.data
)
reader = AirbyteHubspotReader(
config=hubspot_config, record_handler=handle_record
)
Lazy loads
The reader.load_data endpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By using reader.lazy_load_data instead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.
Incremental loads
This loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):
reader = AirbyteHubspotReader(config={...})
documents = reader.load_data(stream_name="products")
current_state = reader.last_state # can be pickled away or stored otherwise
updated_documents = reader.load_data(
stream_name="products", state=current_state
) # only loads documents that were updated since last time
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.