Generate Article Recommendations from Embeddings
A notebook building an embedding-based nearest-neighbor recommender for news articles, with cached embeddings and t-SNE visualization.
Why it matters
Leverage AI embeddings and nearest neighbor search to identify and recommend similar articles based on content. This asset helps discover related news, books, or web pages.
Outcomes
What it gets done
Generate embeddings for article descriptions.
Calculate similarity between articles using embeddings.
Recommend articles closest to a given source article.
Visualize article clusters based on embeddings.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-recommendationusingembeddings | bash Steps
Steps in the chain
Overview
Recommendation using embeddings and nearest neighbor search
A notebook building an embedding-based nearest-neighbor recommender for news articles, with a persistent embedding cache and t-SNE/PCA visualization of the embedding space. Use it as a reference for bootstrapping a similar-items recommender from text embeddings alone, especially for cold-start items with no user data.
What it does
Builds a similar-item recommender using embeddings and nearest-neighbor search, applied to AG's corpus of news articles: given one article, find the others most similar to it. Embeddings are cached in a dictionary mapping (text, model) tuples to their embedding vector, persisted as a Python pickle file so they're never recomputed, and never re-paid for, once generated. Recommendation itself follows three steps: embed every article's description, compute the distance from a source article to every other article, and surface the closest matches. On a Tony Blair article, four of the five recommendations explicitly mention Tony Blair and the fifth is a topically related London climate-change piece; on an NVIDIA chipset-security article, the top recommendation sits distinctly closer than the rest (0.11 vs. 0.14+) and is another PC World article about improving computer security. Two appendices extend the core idea: embeddings as one signal among many in a more sophisticated recommender that also uses item popularity or click data - particularly valuable for cold-started items with no user data yet - and visualizing the article embeddings by compressing them from their full dimensionality down to 2-3 dimensions with t-SNE or PCA, noting that t-SNE isn't deterministic, so results vary run to run. The compressed visualization still clusters articles well by category with no label information at all, and most visible outliers turn out to be mislabeled source data rather than bad embeddings. Compression does lose information, though: for the Tony Blair example, the 5 nearest neighbors in full-dimensional space aren't necessarily the closest points once compressed to 2D, while for the chipset example 4 of 5 stay nearest even after compression. An optional interactive 3D plot via chart_from_components_3D is available by recomputing t-SNE with n_components=3.
When to use - and when NOT to
Use it as a reference for building a similar-items recommender directly from text embeddings and nearest-neighbor distance - a fast way to bootstrap recommendations, especially for items with no click or popularity history - rather than as a replacement for a full multi-signal recommendation model once that data exists.
Inputs and outputs
Input is a corpus of text items, here news article descriptions, and a source item to find neighbors for. Output is a ranked list of the nearest items by embedding distance, an on-disk embedding cache to avoid recomputation, and optional 2D/3D visualizations of the embedding space via t-SNE or PCA.
Integrations
Built on OpenAI's embeddings API for generating item vectors, a Python pickle-based cache keyed by (text, model), and t-SNE/PCA for dimensionality reduction and charting.
Who it's for
For developers building a first-pass similar-items recommender from existing content, especially where user interaction data is sparse or absent - new catalog items, cold-start scenarios - and a pure content-similarity signal is the fastest path to a useful recommendation.
Source README
Recommendation using embeddings and nearest neighbor search
Recommendations are widespread across the web.
- 'Bought that item? Try these similar items.'
- 'Enjoy that book? Try these similar titles.'
- 'Not the help page you were looking for? Try these similar pages.'
This notebook demonstrates how to use embeddings to find similar items to recommend. In particular, we use AG's corpus of news articles as our dataset.
Our model will answer the question: given an article, what other articles are most similar to it?
2. Load data
Next, let's load the AG news data and see what it looks like.
Let's take a look at those same examples, but not truncated by ellipses.
3. Build cache to save embeddings
Before getting embeddings for these articles, let's set up a cache to save the embeddings we generate. In general, it's a good idea to save your embeddings so you can reuse them later. If you don't save them, you'll pay each time you recompute them.
The cache is a dictionary that maps tuples of (text, model) to an embedding, which is a list of floats. The cache is saved as a Python pickle file.
Let's check that it works by getting an embedding.
4. Recommend similar articles based on embeddings
To find similar articles, let's follow a three-step plan:
- Get the similarity embeddings of all the article descriptions
- Calculate the distance between a source title and all other articles
- Print out the other articles closest to the source title
5. Example recommendations
Let's look for articles similar to first one, which was about Tony Blair.
Pretty good! 4 of the 5 recommendations explicitly mention Tony Blair and the fifth is an article from London about climate change, topics that might be often associated with Tony Blair.
Let's see how our recommender does on the second example article about NVIDIA's new chipset with more security.
From the printed distances, you can see that the #1 recommendation is much closer than all the others (0.11 vs 0.14+). And the #1 recommendation looks very similar to the starting article - it's another article from PC World about increasing computer security. Pretty good!
Appendix: Using embeddings in more sophisticated recommenders
A more sophisticated way to build a recommender system is to train a machine learning model that takes in tens or hundreds of signals, such as item popularity or user click data. Even in this system, embeddings can be a very useful signal into the recommender, especially for items that are being 'cold started' with no user data yet (e.g., a brand new product added to the catalog without any clicks yet).
Appendix: Using embeddings to visualize similar articles
To get a sense of what our nearest neighbor recommender is doing, let's visualize the article embeddings. Although we can't plot the 2048 dimensions of each embedding vector, we can use techniques like t-SNE or PCA to compress the embeddings down into 2 or 3 dimensions, which we can chart.
Before visualizing the nearest neighbors, let's visualize all of the article descriptions using t-SNE. Note that t-SNE is not deterministic, meaning that results may vary from run to run.
As you can see in the chart above, even the highly compressed embeddings do a good job of clustering article descriptions by category. And it's worth emphasizing: this clustering is done with no knowledge of the labels themselves!
Also, if you look closely at the most egregious outliers, they are often due to mislabeling rather than poor embedding. For example, the majority of the blue World points in the green Sports cluster appear to be Sports stories.
Next, let's recolor the points by whether they are a source article, its nearest neighbors, or other.
Looking at the 2D chart above, we can see that the articles about Tony Blair are somewhat close together inside of the World news cluster. Interestingly, although the 5 nearest neighbors (red) were closest in high dimensional space, they are not the closest points in this compressed 2D space. Compressing the embeddings down to 2 dimensions discards much of their information, and the nearest neighbors in the 2D space don't seem to be as relevant as those in the full embedding space.
For the chipset security example, the 4 closest nearest neighbors in the full embedding space remain nearest neighbors in this compressed 2D visualization. The fifth is displayed as more distant, despite being closer in the full embedding space.
Should you want to, you can also make an interactive 3D plot of the embeddings with the function chart_from_components_3D. (Doing so will require recomputing the t-SNE components with n_components=3.)
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.