Integrate OpenAI Embeddings with Tair Vector Database
A notebook using Tair, Alibaba Cloud's in-memory vector database, to store and nearest-neighbor search OpenAI embeddings.
Why it matters
Leverage Tair as a high-performance vector database to store and query OpenAI embeddings for efficient similarity searches.
Outcomes
What it gets done
Store OpenAI-generated embeddings in Tair.
Perform nearest neighbor searches using Tair's vector capabilities.
Convert text queries to embeddings using the OpenAI API.
Index and retrieve vector data for applications.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-gettingstartedwithtairandopenai | bash Steps
Steps in the chain
Overview
Using Tair as a vector database for OpenAI embeddings
A notebook storing and nearest-neighbor searching OpenAI embeddings in Tair, Alibaba Cloud's Redis-compatible in-memory vector database, using HNSW indexing. Use it as a reference for vector search on Tair, particularly for teams already running Redis-compatible infrastructure who want to avoid a separate vector database.
What it does
Walks through storing and searching OpenAI embeddings in Tair, Alibaba Cloud's Redis-compatible, in-memory database service. The end-to-end flow: use precomputed OpenAI embeddings, store them in a cloud Tair instance, convert a raw text query into an embedding via the OpenAI API, and run nearest-neighbor search against the stored vectors with Tair. Tair's vector capability, TairVector, is an in-house data structure supporting two indexing algorithms - Hierarchical Navigable Small World (HNSW) and Flat Search - and multiple distance functions: Euclidean, inner product, and Jaccard. Compared to traditional vector retrieval services, it keeps all data in memory with real-time index updates to cut read/write latency, uses an optimized in-memory data structure for better storage utilization, and works as a simple, dependency-free out-of-the-box structure. Tair also offers persistent memory-optimized instances built on non-volatile memory storage, cutting costs by roughly 30% while keeping data persistence and near-in-memory performance - the underlying service is used across government, finance, manufacturing, healthcare, and internet-scale workloads. The notebook creates two Tair indexes, one for title_vector and one for content_vector, each storing objects as a key containing a vector plus multiple attribute keys, then loads a set of precomputed Wikipedia-article embeddings so the reader doesn't need to recompute them. Search queries can target either index via a vector_name parameter, and since the precomputed embeddings were generated with text-embedding-3-small, queries must use that same model to stay comparable.
When to use - and when NOT to
Use it as a reference for standing up vector search on Tair, particularly if you're already running Redis-compatible infrastructure and want vector search without adding a separate specialized vector database, rather than for workloads that need a dedicated vector-only service with no Redis compatibility requirement.
Setup requires a provisioned Tair cloud server instance, the tair Python library, and an OpenAI API key, with the connection to the running Tair instance verified by a ping before indexing begins.
Inputs and outputs
Input is precomputed or freshly generated OpenAI embeddings plus a raw text query. Output is the nearest-neighbor search results from Tair, ranked by vector similarity against either the title or content vector index.
Integrations
Built on Tair via the official tair Python library and the OpenAI API for embedding generation, specifically text-embedding-3-small to match the notebook's precomputed dataset.
Who it's for
For developers who want vector search backed by a Redis-compatible, in-memory database, particularly teams already invested in Tair or Redis infrastructure who want to add embedding-based search without introducing a separate vector-database service.
Source README
Using Tair as a vector database for OpenAI embeddings
This notebook guides you step by step on using Tair as a vector database for OpenAI embeddings.
This notebook presents an end-to-end process of:
- Using precomputed embeddings created by OpenAI API.
- Storing the embeddings in a cloud instance of Tair.
- Converting raw text query to an embedding with OpenAI API.
- Using Tair to perform the nearest neighbour search in the created collection.
What is Tair
Tair is a cloud native in-memory database service that is developed by Alibaba Cloud. Tair is compatible with open source Redis and provides a variety of data models and enterprise-class capabilities to support your real-time online scenarios. Tair also introduces persistent memory-optimized instances that are based on the new non-volatile memory (NVM) storage medium. These instances can reduce costs by 30%, ensure data persistence, and provide almost the same performance as in-memory databases. Tair has been widely used in areas such as government affairs, finance, manufacturing, healthcare, and pan-Internet to meet their high-speed query and computing requirements.
Tairvector is an in-house data structure that provides high-performance real-time storage and retrieval of vectors. TairVector provides two indexing algorithms: Hierarchical Navigable Small World (HNSW) and Flat Search. Additionally, TairVector supports multiple distance functions, such as Euclidean distance, inner product, and Jaccard distance. Compared with traditional vector retrieval services, TairVector has the following advantages:
- Stores all data in memory and supports real-time index updates to reduce latency of read and write operations.
- Uses an optimized data structure in memory to better utilize storage capacity.
- Functions as an out-of-the-box data structure in a simple and efficient architecture without complex modules or dependencies.
Deployment options
- Using Tair Cloud Vector Database. Click here to fast deploy it.
Prerequisites
For the purposes of this exercise we need to prepare a couple of things:
- Tair cloud server instance.
- The 'tair' library to interact with the tair database.
- An OpenAI API key.
Install requirements
This notebook obviously requires the openai and tair packages, but there are also some other additional libraries we will use. The following command installs them all:
Prepare your OpenAI API key
The OpenAI API key is used for vectorization of the documents and queries.
If you don't have an OpenAI API key, you can get one from https://beta.openai.com/account/api-keys.
Once you get your key, please add it by getpass.
Connect to Tair
First add it to your environment variables.
Connecting to a running instance of Tair server is easy with the official Python library.
We can test the connection by ping:
The downloaded file has to then be extracted:
Create Index
Tair stores data in indexes where each object is described by one key. Each key contains a vector and multiple attribute_keys.
We will start with creating two indexes, one for title_vector and one for content_vector, and then we will fill it with our precomputed embeddings.
Load data
In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.
Search data
Once the data is put into Tair we will start querying the collection for the closest vectors. We may provide an additional parameter vector_name to switch from title to content based search. Since the precomputed embeddings were created with text-embedding-3-small OpenAI model, we also have to use it during search.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.