Query Movies with Embeddings and Metadata Filters
Notebook combining OpenAI embeddings with Zilliz metadata filtering to search over 8,000+ movie descriptions.
Why it matters
Leverage OpenAI embeddings and Zilliz vector database to find relevant movies based on descriptions and filter by metadata like release year or genre.
Outcomes
What it gets done
Generate embeddings for movie descriptions using OpenAI.
Store movie data and embeddings in Zilliz.
Perform filtered searches using natural language descriptions and metadata.
Retrieve and display relevant movie search results.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-filteredsearchwithzillizandopenai | bash Steps
Steps in the chain
Overview
Filtered Search with Zilliz and OpenAI
A notebook embedding movie descriptions with OpenAI and searching them in Zilliz with metadata filters on type, release year, and rating alongside vector similarity. Use when semantic search needs structured metadata filters on top of similarity search. Not needed if you're not using Zilliz as the vector store or don't need metadata filtering.
What it does
This notebook embeds movie descriptions with OpenAI and stores and searches them in Zilliz, a vector database, adding metadata filtering to narrow results beyond pure similarity search. The dataset is HuggingFace's HuggingLearners netflix-shows dataset - a little over 8,000 movie entries, each with a description plus title, type, release_year, and rating metadata. It installs openai for embedding calls, pymilvus for talking to the Zilliz server, datasets for pulling the dataset, and tqdm for progress bars, then configures the Zilliz connection (URI, USER, PASSWORD, COLLECTION_NAME, DIMENSION, INDEX_PARAM, QUERY_PARAM, BATCH_SIZE) alongside an OpenAI API key and OPENAI_ENGINE embedding model.
Each description is embedded first, then inserted: the notebook iterates through all entries, builds batches, and inserts a batch once it hits the configured BATCH_SIZE, inserting whatever remains after the loop finishes. Each inserted record carries its title, type, release year, and rating alongside the embedding. A query then takes a text description to search for plus a boolean filter expression (for example, restricting results to a given type or release year) - full filter-expression syntax is documented separately in Milvus's boolean-expression docs. The query first prints the description and filter it was run with, then for each returned result prints the score, title, type, release year, rating, and description.
When to use - and when NOT to
Use it when you need semantic search over text narrowed by structured metadata filters rather than similarity alone - for example, finding movies similar to a description but only TV shows released after a given year. It is not a fit if you don't need metadata filtering on top of vector search, or aren't using Zilliz as the vector store.
Inputs and outputs
Input is openai, pymilvus, datasets, and tqdm installed; a Zilliz account and database already set up, with its URI, username, and password on hand; an OpenAI API key and embedding engine; and collection settings (COLLECTION_NAME, DIMENSION, INDEX_PARAM, QUERY_PARAM, BATCH_SIZE). Output is a populated Zilliz collection of movie embeddings plus metadata, and per-query results showing score, title, type, release year, rating, and description for each match.
Integrations
Uses OpenAI for embeddings and Zilliz, via pymilvus, as the vector store with boolean metadata filtering; the source dataset comes from HuggingFace Datasets.
Who it's for
Developers building semantic search that needs metadata filters alongside vector similarity - not just finding similar text, but finding similar text that also matches specific structured conditions.
Source README
Filtered Search with Zilliz and OpenAI
Finding your next movie
In this notebook we will be going over generating embeddings of movie descriptions with OpenAI and using those embeddings within Zilliz to find relevant movies. To narrow our search results and try something new, we are going to be using filtering to do metadata searches. The dataset in this example is sourced from HuggingFace datasets, and contains a little over 8 thousand movie entries.
Lets begin by first downloading the required libraries for this notebook:
openaiis used for communicating with the OpenAI embedding servicepymilvusis used for communicating with the Zilliz serverdatasetsis used for downloading the datasettqdmis used for the progress bars
To get Zilliz up and running take a look here. With your account and database set up, proceed to set the following values:
- URI: The URI your database is running on
- USER: Your database username
- PASSWORD: Your database password
- COLLECTION_NAME: What to name the collection within Zilliz
- DIMENSION: The dimension of the embeddings
- OPENAI_ENGINE: Which embedding model to use
- openai.api_key: Your OpenAI account key
- INDEX_PARAM: The index settings to use for the collection
- QUERY_PARAM: The search parameters to use
- BATCH_SIZE: How many texts to embed and insert at once
Dataset
With Zilliz up and running we can begin grabbing our data. Hugging Face Datasets is a hub that holds many different user datasets, and for this example we are using HuggingLearners's netflix-shows dataset. This dataset contains movies and their metadata pairs for over 8 thousand movies. We are going to embed each description and store it within Zilliz along with its title, type, release_year and rating.
Insert the Data
Now that we have our data on our machine we can begin embedding it and inserting it into Zilliz. The embedding function takes in text and returns the embeddings in a list format.
This next step does the actual inserting. We iterate through all the entries and create batches that we insert once we hit our set batch size. After the loop is over we insert the last remaining batch if it exists.
Query the Database
With our data safely inserted into Zilliz, we can now perform a query. The query takes in a tuple of the movie description you are searching for and the filter to use. More info about the filter can be found here. The search first prints out your description and filter expression. After that for each result we print the score, title, type, release year, rating and description of the result movies.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.