Load Snowflake Data into LlamaIndex
Query Snowflake and load results as documents into LlamaIndex via SQLAlchemy.
Why it matters
Effortlessly ingest data from Snowflake into LlamaIndex for advanced AI applications. This asset streamlines the process of querying your Snowflake database and transforming the results into Document objects.
Outcomes
What it gets done
Connect to Snowflake using SQLAlchemy or direct credentials.
Execute custom SQL queries against your Snowflake tables.
Extract query results as LlamaIndex Document objects.
Prepare data for ingestion into LlamaIndex indexes like GPTSQLStructStoreIndex.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-snowflake | bash Overview
Snowflake Loader
A LlamaIndex reader that runs a SQL query against Snowflake via SQLAlchemy and returns the matching rows as documents. Use when you already store data in Snowflake and want a specific query's results loaded for indexing, not for writing back to Snowflake.
What it does
The Snowflake Loader connects to Snowflake, using SQLAlchemy under the hood, so a user can specify a query and get back Document objects corresponding to the query results. It is built to pass the resulting documents into a GPTSQLStructStoreIndex from LlamaIndex, though the documents work with any LlamaIndex index.
SnowflakeReader supports two initialization paths. The first passes an already-created SQLAlchemy engine object directly to the reader, so any existing connection setup is reused as-is. The second passes the connection parameters directly to the reader - account, user, database, schema, warehouse, an optional role, and an optional proxy setting - and the reader builds the connection itself. Either way, load_data then takes a SQL query string and returns the matching rows as documents.
When to use - and when NOT to
Use it when you already have data in Snowflake and want to bring the result of a specific SQL query into a LlamaIndex index for retrieval or question-answering, and you either have an existing SQLAlchemy engine to reuse or are comfortable passing Snowflake connection parameters directly. Do not use it for writing data back to Snowflake - it is a read loader, taking a SELECT-style query and returning documents, not a general database client.
Capabilities
load_data runs a SQL query against Snowflake (via a passed-in SQLAlchemy engine or direct connection parameters) and returns the matching rows as LlamaIndex Document objects.
How to install
pip install llama-index-readers-snowflake
Requires either an existing SQLAlchemy engine for Snowflake, or account, user, password, database, schema, and warehouse connection parameters (role and proxy settings are optional).
Who it's for
Developers who already store data in Snowflake and want the result of a specific query loaded into a LlamaIndex pipeline without hand-writing a separate database-to-document conversion step.
Source README
Snowflake Loader
pip install llama-index-readers-snowflake
This loader connects to Snowflake (using SQLAlchemy under the hood). The user specifies a query and extracts Document objects corresponding to the results. You can use this loader to easily connect to a database on Snowflake and pass the documents into a GPTSQLStructStoreIndex from LlamaIndex.
Usage
Option 1: Pass your own SQLAlchemy Engine object of the database connection
Here's an example usage of the SnowflakeReader.
from llama_index.readers.snowflake import SnowflakeReader
reader = SnowflakeReader(
engine=your_sqlalchemy_engine,
)
query = "SELECT * FROM your_table"
documents = reader.load_data(query=query)
Option 2: Pass the required parameters to esstablish Snowflake connection
Here's an example usage of the SnowflakeReader.
from llama_index.readers.snowflake import SnowflakeReader
reader = SnowflakeReader(
account="your_account",
user="your_user",
[REDACTED],
database="your_database",
schema="your_schema",
warehouse="your_warehouse",
role="your_role", # Optional role setting
proxy="http://proxusername:proxypassword@myproxy:port", # Optional proxy setting
)
query = "SELECT * FROM your_table"
documents = reader.load_data(query=query)
Author
This loader is designed to be used as a way to load data into LlamaIndex.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.