Vector Search with OpenAI Embeddings in AnalyticDB
OpenAI cookbook for using Alibaba Cloud's AnalyticDB as a vector database for OpenAI embeddings and nearest-neighbor search.
Why it matters
Leverage AnalyticDB as a high-performance vector database for your OpenAI embeddings. This asset enables efficient nearest neighbor searches on your data, powered by cloud-native vector compute.
Outcomes
What it gets done
Store OpenAI embeddings in AnalyticDB.
Convert text queries to embeddings using OpenAI API.
Perform nearest neighbor searches within AnalyticDB.
Utilize AnalyticDB's PostgreSQL compatibility for data management.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-gettingstartedwithanalyticdbandopenai | bash Steps
Steps in the chain
Overview
Using AnalyticDB as a vector database for OpenAI embeddings
OpenAI cookbook guide to using Alibaba Cloud's AnalyticDB as a PostgreSQL-compatible vector store for OpenAI embeddings, covering storage of precomputed title/content vectors and nearest-neighbor search with the text-embedding-3-small model. Use when you need a PostgreSQL-compatible, cloud-scale vector database for OpenAI embeddings with OLAP and scalar-filtering capability alongside nearest-neighbor search.
What it does
This OpenAI cookbook notebook is a step-by-step guide to using AnalyticDB - Alibaba Cloud's managed, PostgreSQL-compatible distributed vector database - as a vector store for OpenAI embeddings. It walks through an end-to-end process: taking precomputed embeddings created via the OpenAI API, storing them in a cloud instance of AnalyticDB, converting a raw text query into an embedding with the OpenAI API, and running a nearest-neighbor search over the stored collection to find the closest matches. AnalyticDB is described as fully PostgreSQL-syntax compatible, cloud-native, and capable of scaling to billions of vectors, with indexing algorithms, structured and unstructured data support, real-time updates, multiple distance metrics, scalar filtering, and time-travel search, backed by full OLAP functionality and an SLA for production use. The guide has you provision an AnalyticDB cloud server instance, install the openai and psycopg2 (or another PostgreSQL client library) packages, and set your OpenAI API key as the OPENAI_API_KEY environment variable before connecting via the official Python library. Data is organized into a relation called articles, where each object is described by both a title vector and a content vector; the guide creates this relation with vector indexes on both fields, loads precomputed Wikipedia article embeddings so you don't have to recompute them yourself, and then queries the collection - optionally passing a vector_name parameter to search against title or content vectors - using the same text-embedding-3-small OpenAI model that generated the original embeddings, since query and stored vectors must come from the same embedding model to be comparable.
When to use - and when NOT to
Use this guide when you need a PostgreSQL-compatible, cloud-native vector store that can scale to billions of vectors with full OLAP capability alongside vector search - for example if your infrastructure is already on Alibaba Cloud or you need scalar filtering and time-travel search alongside nearest-neighbor lookup. It is not a guide to building a full RAG or question-answering application (see the companion Langchain + AnalyticDB notebook for that) - this notebook covers only storage and nearest-neighbor retrieval of precomputed embeddings.
Inputs and outputs
Input is precomputed OpenAI embeddings (title and content vectors) plus a raw text search query. Output is a ranked list of nearest-neighbor matches from the AnalyticDB articles relation, retrieved by comparing the query's text-embedding-3-small embedding against the stored title or content vectors.
Integrations
Built on Alibaba Cloud AnalyticDB for PostgreSQL, the OpenAI API (text-embedding-3-small for vectorization), and the psycopg2 Python PostgreSQL client library (or any equivalent PostgreSQL client).
Who it's for
Developers who need a PostgreSQL-compatible, horizontally scalable cloud vector database for OpenAI embeddings - particularly those already using or considering Alibaba Cloud infrastructure - and want a minimal working example of storage plus nearest-neighbor search before building a full application on top.
Source README
Using AnalyticDB as a vector database for OpenAI embeddings
This notebook guides you step by step on using AnalyticDB as a vector database for OpenAI embeddings.
This notebook presents an end-to-end process of:
- Using precomputed embeddings created by OpenAI API.
- Storing the embeddings in a cloud instance of AnalyticDB.
- Converting raw text query to an embedding with OpenAI API.
- Using AnalyticDB to perform the nearest neighbour search in the created collection.
What is AnalyticDB
AnalyticDB is a high-performance distributed vector database. Fully compatible with PostgreSQL syntax, you can effortlessly utilize it. AnalyticDB is Alibaba Cloud managed cloud-native database with strong-performed vector compute engine. Absolute out-of-box experience allow to scale into billions of data vectors processing with rich features including indexing algorithms, structured & non-structured data features, realtime update, distance metrics, scalar filtering, time travel searches etc. Also equipped with full OLAP database functionality and SLA commitment for production usage promise;
Deployment options
- Using AnalyticDB Cloud Vector Database. Click here to fast deploy it.
Prerequisites
For the purposes of this exercise we need to prepare a couple of things:
- AnalyticDB cloud server instance.
- The 'psycopg2' library to interact with the vector database. Any other postgresql client library is ok.
- An OpenAI API key.
We might validate if the server was launched successfully by running a simple curl command:
Install requirements
This notebook obviously requires the openai and psycopg2 packages, but there are also some other additional libraries we will use. The following command installs them all:
Prepare your OpenAI API key
The OpenAI API key is used for vectorization of the documents and queries.
If you don't have an OpenAI API key, you can get one from https://beta.openai.com/account/api-keys.
Once you get your key, please add it to your environment variables as OPENAI_API_KEY.
Connect to AnalyticDB
First add it to your environment variables. or you can just change the "psycopg2.connect" parameters below
Connecting to a running instance of AnalyticDB server is easy with the official Python library:
We can test the connection by running any available method:
The downloaded file has to be then extracted:
Index data
AnalyticDB stores data in relation where each object is described by at least one vector. Our relation will be called articles and each object will be described by both title and content vectors. \
We will start with creating a relation and create a vector index on both title and content, and then we will fill it with our precomputed embeddings.
Load data
In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.
Search data
Once the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter vector_name to switch from title to content based search. Since the precomputed embeddings were created with text-embedding-3-small OpenAI model we also have to use it during search.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.