Vector Search with OpenAI Embeddings in AnalyticDB

Using AnalyticDB as a vector database for OpenAI embeddings

This notebook guides you step by step on using AnalyticDB as a vector database for OpenAI embeddings.

This notebook presents an end-to-end process of:

1. Using precomputed embeddings created by OpenAI API.
2. Storing the embeddings in a cloud instance of AnalyticDB.
3. Converting raw text query to an embedding with OpenAI API.
4. Using AnalyticDB to perform the nearest neighbour search in the created collection.

What is AnalyticDB

AnalyticDB is a high-performance distributed vector database. Fully compatible with PostgreSQL syntax, you can effortlessly utilize it. AnalyticDB is Alibaba Cloud managed cloud-native database with strong-performed vector compute engine. Absolute out-of-box experience allow to scale into billions of data vectors processing with rich features including indexing algorithms, structured & non-structured data features, realtime update, distance metrics, scalar filtering, time travel searches etc. Also equipped with full OLAP database functionality and SLA commitment for production usage promise;

Deployment options

• Using AnalyticDB Cloud Vector Database. Click here to fast deploy it.

Prerequisites

For the purposes of this exercise we need to prepare a couple of things:

1. AnalyticDB cloud server instance.
2. The 'psycopg2' library to interact with the vector database. Any other postgresql client library is ok.
3. An OpenAI API key.

We might validate if the server was launched successfully by running a simple curl command:

Install requirements

This notebook obviously requires the openai and psycopg2 packages, but there are also some other additional libraries we will use. The following command installs them all:

Prepare your OpenAI API key

The OpenAI API key is used for vectorization of the documents and queries.

If you don't have an OpenAI API key, you can get one from https://beta.openai.com/account/api-keys.

Once you get your key, please add it to your environment variables as OPENAI_API_KEY.

Connect to AnalyticDB

First add it to your environment variables. or you can just change the "psycopg2.connect" parameters below

Connecting to a running instance of AnalyticDB server is easy with the official Python library:

We can test the connection by running any available method:

The downloaded file has to be then extracted:

Index data

AnalyticDB stores data in relation where each object is described by at least one vector. Our relation will be called articles and each object will be described by both title and content vectors. \

We will start with creating a relation and create a vector index on both title and content, and then we will fill it with our precomputed embeddings.

Load data

In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.

Search data

Once the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter vector_name to switch from title to content based search. Since the precomputed embeddings were created with text-embedding-3-small OpenAI model we also have to use it during search.