Generate Embeddings from Large Datasets
Generate and save OpenAI embeddings for a large dataset, using 1,000 Amazon fine-food reviews as the example.
Why it matters
Process large text datasets to generate vector embeddings for downstream machine learning tasks. This asset is ideal for preparing unstructured data for analysis or model training.
Outcomes
What it gets done
Load and preprocess text data from a specified dataset.
Generate vector embeddings for combined review summaries and text.
Save generated embeddings for efficient future reuse.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-getembeddingsfromdataset | bash Steps
Steps in the chain
Overview
Get embeddings from dataset
An OpenAI Cookbook notebook demonstrating how to compute and save embeddings for a large tabular dataset, using Amazon fine-food reviews as the worked example. Use it as a starting template for embedding a tabular dataset with text fields. Built on a small 1,000-row subset - scaling up needs its own batching and rate-limit handling.
What it does
This notebook demonstrates getting embeddings from a large dataset, using a subset of the Amazon fine-food reviews dataset (568,454 total reviews, of which this example uses the 1,000 most recent) as the worked example. Each review's ProductId, UserId, Score, review title (Summary), and review body (Text) get combined - the summary and text are joined into one combined string - and the model encodes that combined text into a single vector embedding per review.
When to use - and when NOT to
Use this as a starting template whenever you need to turn a tabular dataset with free-text fields into embeddings for downstream tasks like search, clustering, or classification. It's built around a small illustrative subset (1,000 rows) rather than the full 568,454-review dataset, so scaling up would mean revisiting how embeddings are batched and rate-limited.
Inputs and outputs
The dataset loads with pandas, and the combined summary-plus-text field is what actually gets embedded, not the raw fields separately - this keeps title context and review body together in one vector rather than losing the title's signal. Once computed, the embeddings are saved for future reuse rather than recomputed on every run.
Integrations
Running the notebook requires pandas, openai, transformers, plotly, matplotlib, scikit-learn, torch (a transformers dependency), torchvision, and scipy installed.
Who it's for
Developers who need a basic, reusable pattern for embedding a real-world tabular dataset with text fields, as a foundation for later steps like search, clustering, or classification on the same data.
Source README
Get embeddings from dataset
This notebook gives an example on how to get embeddings from a large dataset.
1. Load the dataset
The dataset used in this example is fine-food reviews from Amazon. The dataset contains a total of 568,454 food reviews Amazon users left up to October 2012. We will use a subset of this dataset, consisting of 1,000 most recent reviews for illustration purposes. The reviews are in English and tend to be positive or negative. Each review has a ProductId, UserId, Score, review title (Summary) and review body (Text).
We will combine the review summary and review text into a single combined text. The model will encode this combined text and it will output a single vector embedding.
To run this notebook, you will need to install: pandas, openai, transformers, plotly, matplotlib, scikit-learn, torch (transformer dep), torchvision, and scipy.
2. Get embeddings and save them for future reuse
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.