Integrate Weaviate with OpenAI for Hybrid Search

Using Weaviate with OpenAI vectorize module for Hybrid Search

This notebook is prepared for a scenario where:

• Your data is not vectorized
• You want to run Hybrid Search (learn more) on your data
• You want to use Weaviate with the OpenAI module (text2vec-openai), to generate vector embeddings for you.

This notebook takes you through a simple flow to set up a Weaviate instance, connect to it (with OpenAI API key), configure data schema, import data (which will automatically generate vector embeddings for your data), and run hybrid search (mixing of vector and BM25 search).

This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.

What is Weaviate

Weaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.

Weaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more here.

Weaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.

Deployment options

Whatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:

• Self-hosted - you can deploy Weaviate with docker locally, or any server you want.
• SaaS - you can use Weaviate Cloud Service (WCS) to host your Weaviate instances.
• Hybrid-SaaS - you can deploy Weaviate in your own private Cloud Service

Programming languages

Weaviate offers four client libraries, which allow you to communicate from your apps:

• Python
• JavaScript
• Java
• Go

Additionally, Weaviate has a REST layer. Basically you can call Weaviate from any language that supports REST requests.

Demo Flow

The demo flow is:

• Prerequisites Setup: Create a Weaviate instance and install required libraries
• Connect: Connect to your Weaviate instance
• Schema Configuration: Configure the schema of your data
  • Note: Here we can define which OpenAI Embedding Model to use
  • Note: Here we can configure which properties to index
• Import data: Load a demo dataset and import it into Weaviate
  • Note: The import process will automatically index your data - based on the configuration in the schema
  • Note: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you
• Run Queries: Query
  • Note: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.

OpenAI Module in Weaviate

All Weaviate instances come equipped with the text2vec-openai module.

This module is responsible for handling vectorization during import (or any CRUD operations) and when you run a query.

No need to manually vectorize data

This is great news for you. With text2vec-openai you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.

All you need to do is:

1. provide your OpenAI API Key - when you connected to the Weaviate Client
2. define which OpenAI vectorizer to use in your Schema

Prerequisites

Before we start this project, we need setup the following:

• create a Weaviate instance
• install libraries
  • weaviate-client
  • datasets
  • apache-beam
• get your OpenAI API key

===========================================================

Create a Weaviate instance

To create a Weaviate instance we have 2 options:

1. (Recommended path) Weaviate Cloud Service - to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.
2. Install and run Weaviate locally with Docker.

Option 1 - WCS Installation Steps

Use Weaviate Cloud Service (WCS) to create a free Weaviate cluster.

1. create a free account and/or login to WCS
2. create a Weaviate Cluster with the following settings:
   • Sandbox: Sandbox Free
   • Weaviate Version: Use default (latest)
   • OIDC Authentication: Disabled
3. your instance should be ready in a minute or two
4. make a note of the Cluster Id. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: https://your-project-name.weaviate.network

Option 2 - local Weaviate instance with Docker

Install and run Weaviate locally with Docker.

1. Download the ./docker-compose.yml file
2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: docker-compose up -d
3. Once this is ready, your instance should be available at http://localhost:8080

Note. To shut down your docker instance you can call: docker-compose down

Learn more

To learn more, about using Weaviate with Docker see the installation documentation.

===========================================================

Install required libraries

Before running this project make sure to have the following libraries:

Weaviate Python client

The Weaviate Python client allows you to communicate with your Weaviate instance from your Python project.

datasets & apache-beam

To load sample data, you need the datasets library and its' dependency apache-beam.

===========================================================

Prepare your OpenAI API key

The OpenAI API key is used for vectorization of your data at import, and for running queries.

If you don't have an OpenAI API key, you can get one from https://beta.openai.com/account/api-keys.

Once you get your key, please add it to your environment variables as OPENAI_API_KEY.

Connect to your Weaviate instance

In this section, we will:

1. test env variable OPENAI_API_KEY - make sure you completed the step in #Prepare-your-OpenAI-API-key
2. connect to your Weaviate your OpenAI API Key
3. and test the client connection

The client

After this step, the client object will be used to perform all Weaviate-related operations.

Schema

In this section, we will:

1. configure the data schema for your data
2. select OpenAI module

This is the second and final step, which requires OpenAI specific configuration.
After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.

What is a schema

In Weaviate you create schemas to capture each of the entities you will be searching.

A schema is how you tell Weaviate:

• what embedding model should be used to vectorize the data
• what your data is made of (property names and types)
• which properties should be vectorized and indexed

In this cookbook we will use a dataset for Articles, which contains:

• title
• content
• url

We want to vectorize title and content, but not the url.

To vectorize and query the data, we will use text-embedding-3-small.

Import data

In this section we will:

1. load the Simple Wikipedia dataset
2. configure Weaviate Batch import (to make the import more efficient)
3. import the data into Weaviate

Note:

Like mentioned before. We don't need to manually vectorize the data.

The text2vec-openai module will take care of that.

Search Data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors

Learn more about the alpha setting here

Thanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo.