Search Movies with Vector Embeddings and Metadata Filters
Find movies by description with OpenAI embeddings in Milvus, narrowed by metadata filters like type, release year, and rating.
Why it matters
Leverage OpenAI embeddings and Milvus vector search to find relevant movies based on descriptions and filter by metadata like release year and rating.
Outcomes
What it gets done
Generate embeddings for movie descriptions using OpenAI.
Store movie data and embeddings in Milvus.
Perform filtered searches on the Milvus database.
Retrieve and display movie search results with scores and metadata.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-filteredsearchwithmilvusandopenai | bash Steps
Steps in the chain
Overview
Filtered Search with Milvus and OpenAI
Embeds movie descriptions with OpenAI and stores them in Milvus, combining semantic search with metadata filters on type, year, and rating. Use when semantic search alone returns too broad a result set and structured metadata filtering needs to narrow it in the same query.
What it does
This notebook builds a movie-finder that embeds movie descriptions with OpenAI and stores them in Milvus, combined with metadata filtering to narrow search results beyond pure semantic similarity. The dataset is HuggingLearners' netflix-shows dataset from Hugging Face, containing over 8,000 movies with title, type, release_year, and rating metadata alongside each description. After launching a standalone Milvus instance (via the provided docker-compose.yaml) and configuring global variables (host/port, collection name, embedding dimension, OpenAI embedding engine/key, index and query parameters, batch size), each movie description is embedded and inserted into Milvus in batches alongside its metadata fields. Queries take a tuple of a movie description to search for and a boolean filter expression (following Milvus's filter syntax) - the query prints the search description and filter, then for each result shows the similarity score, title, type, release year, rating, and description.
Mechanically, the embedding function accepts text and returns the embeddings as a list; insertion then iterates through every dataset entry, accumulating records into batches and inserting each batch once it reaches the configured batch size, then inserting whatever partial batch remains after the loop finishes so no records are dropped at the end.
When to use - and when NOT to
Use this pattern when semantic search alone returns too broad a result set and you need to narrow results using structured metadata (e.g. only movies, only a specific year range, only a certain rating) alongside the vector similarity match. This combines the strengths of both approaches - Milvus's boolean filter expressions constrain the candidate set before or alongside the vector search, rather than requiring a separate post-filtering step.
Inputs and outputs
Input: a movie description to search for, plus a Milvus boolean filter expression (e.g. constraining type, release_year, or rating). Output: a ranked list of matching movies with similarity score, title, type, release year, rating, and description.
The docker-compose.yaml that launches the standalone Milvus instance ships alongside the notebook itself, so the instance it starts is scoped specifically to this walkthrough rather than a shared or production deployment. The Milvus boolean filter syntax used in the query step is documented separately in Milvus's own filter-expression reference, which the notebook links to directly rather than restating.
Integrations
Requires openai (for embeddings), pymilvus (for the Milvus client), datasets (Hugging Face dataset download), tqdm, and a running Milvus instance (via the provided Docker Compose setup).
Who it's for
Developers building a search or recommendation feature that needs both semantic similarity and structured metadata filtering in the same query, using Milvus as the vector store.
Source README
Filtered Search with Milvus and OpenAI
Finding your next movie
In this notebook we will be going over generating embeddings of movie descriptions with OpenAI and using those embeddings within Milvus to find relevant movies. To narrow our search results and try something new, we are going to be using filtering to do metadata searches. The dataset in this example is sourced from HuggingFace datasets, and contains a little over 8 thousand movie entries.
Lets begin by first downloading the required libraries for this notebook:
openaiis used for communicating with the OpenAI embedding servicepymilvusis used for communicating with the Milvus serverdatasetsis used for downloading the datasettqdmis used for the progress bars
With the required packages installed we can get started. Lets begin by launching the Milvus service. The file being run is the docker-compose.yaml found in the folder of this file. This command launches a Milvus standalone instance which we will use for this test.
With Milvus running we can setup our global variables:
- HOST: The Milvus host address
- PORT: The Milvus port number
- COLLECTION_NAME: What to name the collection within Milvus
- DIMENSION: The dimension of the embeddings
- OPENAI_ENGINE: Which embedding model to use
- openai.api_key: Your OpenAI account key
- INDEX_PARAM: The index settings to use for the collection
- QUERY_PARAM: The search parameters to use
- BATCH_SIZE: How many movies to embed and insert at once
Dataset
With Milvus up and running we can begin grabbing our data. Hugging Face Datasets is a hub that holds many different user datasets, and for this example we are using HuggingLearners's netflix-shows dataset. This dataset contains movies and their metadata pairs for over 8 thousand movies. We are going to embed each description and store it within Milvus along with its title, type, release_year and rating.
Insert the Data
Now that we have our data on our machine we can begin embedding it and inserting it into Milvus. The embedding function takes in text and returns the embeddings in a list format.
This next step does the actual inserting. We iterate through all the entries and create batches that we insert once we hit our set batch size. After the loop is over we insert the last remaning batch if it exists.
Query the Database
With our data safely inserted in Milvus, we can now perform a query. The query takes in a tuple of the movie description you are searching for an the filter to use. More info about the filter can be found here. The search first prints out your description and filter expression. After that for each result we print the score, title, type, release year, rating, and description of the result movies.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.