Integrate OpenAI Embeddings with Qdrant Vector Database
OpenAI cookbook for storing OpenAI embeddings in Qdrant and running payload-filtered nearest-neighbor search.
Why it matters
Leverage OpenAI's embedding capabilities by storing and querying them efficiently using the Qdrant vector database. This asset provides a practical guide for setting up this integration, enabling powerful similarity search on your data.
Outcomes
What it gets done
Connect to a Qdrant instance.
Index OpenAI embeddings into Qdrant collections.
Perform nearest neighbor searches using Qdrant.
Utilize Qdrant's payload filtering with vector search.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-gettingstartedwithqdrantandopenai | bash Steps
Steps in the chain
Overview
Using Qdrant as a vector database for OpenAI embeddings
OpenAI cookbook guide to using Qdrant, a Rust-based vector database with built-in payload filtering, as a store for OpenAI embeddings, covering deployment options, collection creation, and nearest-neighbor search with text-embedding-ada-002. Use when you need a schema-free vector database with efficient, built-in metadata filtering for OpenAI embeddings.
What it does
This OpenAI cookbook notebook is a step-by-step guide to using Qdrant - a high-performance, open-source vector search database written in Rust, offering RESTful and gRPC APIs and an official Python qdrant-client library - as a vector store for OpenAI embeddings. It walks through an end-to-end process: using precomputed embeddings created by the OpenAI API, storing them in a local Qdrant instance, converting a raw text query into an embedding, and running nearest-neighbor search over the created collection. Qdrant stores neural embeddings alongside metadata called "payload," which can be used not just to hold extra attributes on a point but also for filtering - and Qdrant's filtering mechanism is built directly into the vector search phase itself, making filtered search efficient rather than a slow post-search step. It covers Qdrant's deployment options - locally or on-premise with Docker, on Kubernetes via the official Helm chart, or on Qdrant Cloud - and notes you can validate a running server with a simple curl command. After installing the openai and qdrant-client packages and setting OPENAI_API_KEY as an environment variable, the guide connects to the running Qdrant server via the Python client, loads precomputed Wikipedia article embeddings (so you don't have to recompute them with your own credits) from an extracted CSV, and creates a collection called Articles where each point is described by both a title vector and a content vector - noting Qdrant requires no schema setup beforehand, so points can be added with minimal configuration. Searching passes an optional vector_name parameter to switch between title- and content-based search, using the same text-embedding-ada-002 model that produced the stored embeddings.
When to use - and when NOT to
Use this guide when you want a schema-free, high-performance vector store with built-in metadata filtering during search - useful when you need to combine similarity search with attribute filters (like category or date) efficiently. It is not a guide to building a full RAG or QA application (see the companion Langchain + Qdrant notebook for that) - this notebook covers only storage and nearest-neighbor retrieval of precomputed embeddings.
Inputs and outputs
Input is precomputed OpenAI embeddings (title and content vectors) plus a raw text search query. Output is a ranked list of nearest-neighbor matches from the Qdrant Articles collection, retrieved by comparing the query's text-embedding-ada-002 embedding against the stored title or content vectors.
Integrations
Built on Qdrant (local Docker, Kubernetes Helm chart, or Qdrant Cloud deployment; RESTful and gRPC APIs; the official qdrant-client Python library) and the OpenAI API (text-embedding-ada-002) for vectorization.
Who it's for
Developers who want a schema-free, high-performance vector database with built-in payload-based filtering for OpenAI embeddings, and need a minimal working example of storage plus nearest-neighbor search before building a full application.
Source README
Using Qdrant as a vector database for OpenAI embeddings
This notebook guides you step by step on using Qdrant as a vector database for OpenAI embeddings. Qdrant is a high-performant vector search database written in Rust. It offers RESTful and gRPC APIs to manage your embeddings. There is an official Python qdrant-client that eases the integration with your apps.
This notebook presents an end-to-end process of:
- Using precomputed embeddings created by OpenAI API.
- Storing the embeddings in a local instance of Qdrant.
- Converting raw text query to an embedding with OpenAI API.
- Using Qdrant to perform the nearest neighbour search in the created collection.
What is Qdrant
Qdrant is an Open Source vector database that allows storing neural embeddings along with the metadata, a.k.a payload. Payloads are not only available for keeping some additional attributes of a particular point, but might be also used for filtering. Qdrant offers a unique filtering mechanism which is built-in into the vector search phase, what makes it really efficient.
Deployment options
Qdrant might be launched in various ways, depending on the target load on the application it might be hosted:
- Locally or on premise, with Docker containers
- On Kubernetes cluster, with the Helm chart
- Using Qdrant Cloud
Integration
Qdrant provides both RESTful and gRPC APIs which makes integration easy, no matter the programming language you use. However, there are some official clients for the most popular languages available, and if you use Python then the Python Qdrant client library might be the best choice.
We might validate if the server was launched successfully by running a simple curl command:
Install requirements
This notebook obviously requires the openai and qdrant-client packages, but there are also some other additional libraries we will use. The following command installs them all:
Prepare your OpenAI API key
The OpenAI API key is used for vectorization of the documents and queries.
If you don't have an OpenAI API key, you can get one from https://beta.openai.com/account/api-keys.
Once you get your key, please add it to your environment variables as OPENAI_API_KEY by running following command:
Connect to Qdrant
Connecting to a running instance of Qdrant server is easy with the official Python library:
We can test the connection by running any available method:
Load data
In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.
The downloaded file has to be then extracted:
And we can finally load it from the provided CSV file:
Index data
Qdrant stores data in collections where each object is described by at least one vector and may contain an additional metadata called payload. Our collection will be called Articles and each object will be described by both title and content vectors. Qdrant does not require you to set up any kind of schema beforehand, so you can freely put points to the collection with a simple setup only.
We will start with creating a collection, and then we will fill it with our precomputed embeddings.
Search data
Once the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter vector_name to switch from title to content based search. Since the precomputed embeddings were created with text-embedding-ada-002 OpenAI model we also have to use it during search.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.