Load Gong Data with Airbyte
Airbyte Gong Loader is a LlamaIndex reader that loads Gong objects into documents, with support for incremental syncs and custom record handling.
Why it matters
Effortlessly ingest and process your Gong data into LlamaIndex. This asset acts as a bridge, enabling seamless data extraction and integration for further analysis and AI-powered applications.
Outcomes
What it gets done
Connect to Gong via Airbyte for data retrieval.
Extract various Gong objects like 'calls'.
Load data incrementally to manage large datasets.
Configure data extraction with custom parameters.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-airbyte-gong | bash Overview
Airbyte Gong Loader
The Airbyte Gong Loader is a LlamaIndex reader that accesses different Gong objects and loads them into document format. It transforms Gong records into LlamaIndex documents with all fields stored as metadata and JSON text representation by default. You can customize document construction with a record handler function and use lazy loading for large datasets. Use this loader when you need to load Gong data into LlamaIndex document structures. It supports incremental syncs to process only new or updated records, avoiding redundant processing of unchanged data.
What it does
The Airbyte Gong Loader is a LlamaIndex integration that accesses different Gong objects and transforms them into LlamaIndex documents. By default, all fields are stored as metadata and the text is set to the JSON representation of all fields, though you can customize this behavior with a record handler.
When to use - and when NOT to
Use this loader when you need to load Gong data into LlamaIndex document structures. It supports incremental loading to process only new or updated records since your last sync. Do not use this loader if your use case requires direct API access without the LlamaIndex document abstraction layer.
Inputs and outputs
You provide a configuration object containing your Gong access key, access key secret, and a start date in ISO format (e.g., 2020-10-20T00:00:00Z). You also specify a stream name (such as "calls") to indicate which Gong objects to load. Optionally, you can provide a custom record handler function to control how records are transformed into documents, and a state object for incremental loading.
The loader returns a list of LlamaIndex Document objects (or an iterator when using lazy loading). Each document contains the Gong record data as text and metadata. When performing incremental loads, the loader supports loading data incrementally using a state object.
Integrations
This loader uses the Airbyte connector. It is designed specifically for LlamaIndex and produces LlamaIndex Document objects. Configuration follows the Airbyte Gong source specification, with full details available in the Airbyte documentation and the source_gong spec.yaml schema on GitHub.
Who it's for
This tool is built for developers and data engineers who use LlamaIndex as their document processing framework and need to ingest Gong data.
Installation and usage
pip install llama-index-readers-airbyte-gong
from llama_index.readers.airbyte_gong import AirbyteGongReader
gong_config = {
"access_key": "<access key name>",
"access_key_secret": "<access key secret>",
"start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>"
}
reader = AirbyteGongReader(config=gong_config)
documents = reader.load_data(stream_name="calls")
For incremental loading:
reader = AirbyteGongReader(config={...})
documents = reader.load_data(stream_name="calls")
current_state = reader.last_state
updated_documents = reader.load_data(
stream_name="calls", state=current_state
)
Source README
Airbyte Gong Loader
pip install llama-index-readers-airbyte-gong
The Airbyte Gong Loader allows you to access different Gong objects.
Usage
Here's an example usage of the AirbyteGongReader.
from llama_index.readers.airbyte_gong import AirbyteGongReader
gong_config = {
# ...
}
reader = AirbyteGongReader(config=gong_config)
documents = reader.load_data(stream_name="calls")
Configuration
Check out the Airbyte documentation page for details about how to configure the reader.
The JSON schema the config object should adhere to can be found on Github: https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-gong/source_gong/spec.yaml.
The general shape looks like this:
{
"access_key": "<access key name>",
"access_key_secret": "<access key secret>",
"start_date": "<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>",
}
By default all fields are stored as metadata in the documents and the text is set to the JSON representation of all the fields. Construct the text of the document by passing a record_handler to the reader:
def handle_record(record, id):
return Document(
doc_id=id, text=record.data["title"], extra_info=record.data
)
reader = AirbyteGongReader(config=gong_config, record_handler=handle_record)
Lazy loads
The reader.load_data endpoint will collect all documents and return them as a list. If there are a large number of documents, this can cause issues. By using reader.lazy_load_data instead, an iterator is returned which can be consumed document by document without the need to keep all documents in memory.
Incremental loads
This loader supports loading data incrementally (only returning documents that weren't loaded last time or got updated in the meantime):
reader = AirbyteGongReader(config={...})
documents = reader.load_data(stream_name="calls")
current_state = reader.last_state # can be pickled away or stored otherwise
updated_documents = reader.load_data(
stream_name="calls", state=current_state
) # only loads documents that were updated since last time
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.