What does this semantic text search notebook do?

It performs semantic text search over a dataset of reviews by embedding a search query, computing cosine similarity against pre-computed review embeddings, and returning the top N most similar reviews based on meaning rather than exact keyword matches.

When should I use semantic text search with embeddings instead of keyword search?

Use this pattern when you need to search text by meaning rather than exact keyword match, and when you want efficiency at low cost since documents are embedded once and reused for every query—such as surfacing reviews describing delivery failures without needing specific keywords.

What do I need to set up before using this notebook?

You need a pre-computed embeddings dataset from the companion 'Get embeddings from dataset' notebook, which provides the document embeddings that the search process will compare against using OpenAI's embedding model.

How does the search ranking work?

The notebook embeds your query text with the same embedding model used for the documents, then computes cosine similarity between the query embedding and every pre-computed document embedding, returning the closest matches ranked by similarity score.

Prompt Chain

Search Text Semantically Using Embeddings

Name: Semantic Text Search Using Embeddings
Availability: OnlineOnly
Author: OpenAI Cookbook

OpenAI cookbook for semantic review search: embed a query, rank documents by cosine similarity, and return the top matches.

Copy chain

Works with github

OpenAI Cookbook

Own this? Claim it

Spark score

out of 100

Updated 17 days ago

Version 1.0.0

Models

gpt 4o

Add to Favorites

Why it matters

Efficiently search through large text datasets using semantic embeddings. Find the most relevant information quickly and cost-effectively by comparing query embeddings to document embeddings.

Outcomes

What it gets done

Embed search queries and documents.

Calculate cosine similarity between embeddings.

Retrieve top N most similar documents.

Identify specific information, like delivery failures.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/oai-semantictextsearchusingembeddings | bash

Overview

Semantic Text Search Using Embeddings

An OpenAI cookbook for semantic search over reviews: embed a query, rank documents by cosine similarity, and return the top matches. Use for meaning-based search over free-text reviews or documents; not for exact-match or structured filtering needs.

What it does

Semantic Text Search Using Embeddings searches a set of reviews semantically and cheaply by embedding the search query and finding the most similar reviews by cosine similarity against pre-computed document embeddings, returning the top n best matches. The review dataset itself is built in a companion notebook, Get_embeddings_from_dataset.

When to use - and when NOT to

Use it when you need to search free-text reviews or documents by meaning rather than exact keyword match - the notebook's example demonstrates quickly surfacing reviews describing delivery failures purely from a semantic query, without those reviews needing to contain a specific keyword. It notes that a specialized algorithm aimed at faster embedding search can be used to speed up computation at larger scale. It is not a fit for cases needing exact-match or structured filtering rather than semantic similarity.

Inputs and outputs

Input is a search query and a set of pre-computed document embeddings (produced by the companion embeddings notebook). Output is the top n documents ranked by cosine similarity between the query embedding and each document embedding.

Integrations

Depends on the Get_embeddings_from_dataset notebook to produce the embeddings being searched over.

Who it's for

Developers and analysts who need fast, low-cost semantic search over a review or document corpus - for example surfacing a specific category of complaint or feedback without pre-tagging the data.

Source README

Semantic text search using embeddings

We can search through all our reviews semantically in a very efficient manner and at very low cost, by embedding our search query, and then finding the most similar reviews. The dataset is created in the Get_embeddings_from_dataset Notebook.

Here we compare the cosine similarity of the embeddings of the query and the documents, and show top_n best matches.

We can search through these reviews easily. To speed up computation, we can use a special algorithm, aimed at faster search through embeddings.

As we can see, this can immediately deliver a lot of value. In this example we show being able to quickly find the examples of delivery failures.

FAQ

Common questions

Discussion