Classify Text Using Embeddings Without Labeled Data
OpenAI Cookbook notebook classifying review sentiment with zero labeled training data by comparing embeddings to label-description embeddings.
Why it matters
Perform zero-shot text classification by leveraging embeddings. This asset analyzes text and assigns labels without requiring pre-labeled training data, making it ideal for rapid analysis and categorization.
Outcomes
What it gets done
Embed descriptions of potential labels (e.g., positive, negative sentiment).
Compare embeddings of new text samples to label embeddings using cosine similarity.
Assign the most similar label to each text sample.
Optionally calculate a prediction score for thresholding and performance analysis.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-zero-shotclassificationwithembeddings | bash Steps
Steps in the chain
Overview
Zero Shot Classification With Embeddings
Classifies review sentiment with zero labeled data by comparing sample embeddings to embedded class-label descriptions via cosine distance. Use when you need a quick classifier with no labeled training data, only embeddings and label descriptions.
What it does
This notebook classifies the sentiment of product reviews using embeddings and zero labeled training data, on a dataset created in the companion Get_embeddings_from_dataset notebook. Positive sentiment is defined as 4- and 5-star reviews, negative as 1- and 2-star reviews, with 3-star reviews excluded as neutral. The method embeds short descriptions of each class label (e.g., 'positive' and 'negative'), then classifies new samples by comparing the cosine distance between each sample's embedding and the class-label embeddings - the label with the highest similarity is the prediction. A prediction score (the difference between cosine distance to the positive and negative labels) can be used to plot a precision-recall curve, letting you pick a different precision/recall tradeoff by choosing a different threshold.
When to use - and when NOT to
Use this when you need a classifier with no labeled training examples, only embeddings and a short natural-language description of each class. The notebook shows the approach improves further when label descriptions are more descriptive than single words, and further still when using search-style embeddings rather than similarity embeddings. This is not a replacement for a trained classifier when labeled data is available and higher accuracy is required - it is a zero-shot starting point.
Inputs and outputs
Input: text embeddings of reviews (from the Get_embeddings_from_dataset notebook) plus short text descriptions of each class label. Output: a predicted sentiment label per sample (positive/negative) plus a prediction score usable for a precision-recall curve.
Who it's for
Developers who need a working sentiment or category classifier quickly, without labeled training data, and are willing to iterate on label-description wording and embedding type to improve accuracy.
Source README
Zero-shot classification with embeddings
In this notebook we will classify the sentiment of reviews using embeddings and zero labeled data! The dataset is created in the Get_embeddings_from_dataset Notebook.
We'll define positive sentiment to be 4- and 5-star reviews, and negative sentiment to be 1- and 2-star reviews. 3-star reviews are considered neutral and we won't use them for this example.
We will perform zero-shot classification by embedding descriptions of each class and then comparing new samples to those class embeddings.
Zero-Shot Classification
To perform zero shot classification, we want to predict labels for our samples without any training. To do this, we can simply embed short descriptions of each label, such as positive and negative, and then compare the cosine distance between embeddings of samples and label descriptions.
The highest similarity label to the sample input is the predicted label. We can also define a prediction score to be the difference between the cosine distance to the positive and to the negative label. This score can be used for plotting a precision-recall curve, which can be used to select a different tradeoff between precision and recall, by selecting a different threshold.
We can see that this classifier already performs extremely well. We used similarity embeddings, and the simplest possible label name. Let's try to improve on this by using more descriptive label names, and search embeddings.
Using the search embeddings and descriptive names leads to an additional improvement in performance.
As shown above, zero-shot classification with embeddings can lead to great results, especially when the labels are more descriptive than just simple words.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.