Enhance Customer Feedback Analysis with Image RAG
OpenAI cookbook building a multimodal RAG system with Vision, Responses, and file search to analyze image+text customer feedback.
Why it matters
Leverage Retrieval-Augmented Generation (RAG) with OpenAI's Vision and Responses APIs to analyze multimodal customer feedback, combining image and text for deeper insights.
Outcomes
What it gets done
Generate synthetic image and text data for customer feedback.
Integrate image understanding into RAG systems for enhanced context retrieval.
Utilize OpenAI's File Search and Evals API for performance assessment.
Compare text-only vs. text+image RAG performance for sentiment analysis.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/oai-imageunderstandingwithrag | bash Steps
Steps in the chain
Overview
Image Understanding with RAG using OpenAI's Vision & Responses APIs
OpenAI cookbook that builds a multimodal RAG system over synthetic customer feedback combining text and images, using GPT-5, file search, and two parallel vector stores (image-aware vs text-only) to compare retrieval accuracy. Use when building or evaluating a RAG pipeline that must retrieve and reason over both images and text, such as customer feedback or healthcare records.
What it does
This OpenAI cookbook notebook builds a Retrieval-Augmented Generation (RAG) system over multimodal customer feedback data - text reviews plus photos - using OpenAI's Vision and Responses APIs together with GPT-5 and file search. It walks through generating synthetic training/evaluation data (using the Image API for synthetic photos and the Responses API for synthetic text) since real multimodal datasets are costly to collect, then processing and merging a pre-generated synthetic dataset of customer feedback that mixes short text snippets, review images, and combined multimodal entries. It populates OpenAI's built-in vector store with this data for file search, building two parallel vector stores - one with image understanding enabled and one text-only - so the two retrieval paths can be directly compared. Using natural-language queries against both stores (with attribute-based filters), it demonstrates concrete cases where the text-only RAG system is left uncertain - for example, a positive spaghetti review that only exists as a photo, or a negative burnt-pizza review from June - while the image-aware system retrieves the correct visual evidence and gives a more accurate answer, verified by inspecting the retrieved images directly. Finally, it uses the OpenAI Evals API with a string_check criteria (checking outputs against positive, negative, or unclear) to formally score sentiment-analysis accuracy for both the text-only and text+image runs, examines how the added image context (and the resulting increase in total tokens) affects accuracy, performs basic error analysis on the failed examples, and cleans up the created resources at the end.
When to use - and when NOT to
Use this notebook when building or evaluating a RAG pipeline that must reason over both images and text - customer feedback with photos, healthcare records combining scans and clinical notes, or any noisy real-world dataset with incomplete single-modality information. It is not a guide for text-only RAG, nor does it cover training or fine-tuning a custom vision model - it relies entirely on GPT-5's built-in multimodal understanding plus OpenAI's file search and Evals APIs.
Inputs and outputs
Input is a synthetic or real dataset of customer feedback combining text snippets and images. Output is two populated vector stores (image-aware and text-only), natural-language query results with attribute filtering, and Evals API scoring results (positive/negative/unclear classification accuracy) comparing the two retrieval configurations, alongside error analysis of failed cases.
Integrations
Built on OpenAI's Vision API, Responses API (with GPT-5), the Image API for synthetic image generation, built-in vector store and file search for retrieval, and the Evals API for scoring with string_check criteria.
Who it's for
Developers and ML engineers building multimodal RAG systems - for customer experience analysis, healthcare record review, or any domain where visual and textual evidence must be retrieved and reasoned over together - who need a concrete, evaluated example of image-aware versus text-only retrieval.
Source README
Image Understanding with RAG using OpenAI's Vision & Responses APIs
Welcome! This notebook demonstrates how to build a Retrieval-Augmented Generation (RAG) system using OpenAI’s Vision and Responses APIs. It focuses on multimodal data, combining image and text inputs to analyze customer experiences. The system leverages GPT-5 and integrates image understanding with file search to provide context-aware responses.
Multimodal datasets are increasingly common, particularly in domains like healthcare, where records often contain both visual data (e.g. radiology scans) and accompanying text (e.g. clinical notes). Real-world datasets also tend to be noisy, with incomplete or missing information, making it critical to analyze multiple modalities in tandem.
This guide focuses on a customer service use case: evaluating customer feedback that may include photos, and written reviews. You’ll learn how to synthetically generate both image and text inputs, use file search for context retrieval, and apply the Evals API to assess how incorporating image understanding impacts overall performance.
Overview
Table of Contents
- Setup & Dependencies
- Example Generations
- Data Processing
- Load synthetic datasets
- Merge data
- Populating Vector Store
- Upload data for file search
- Set up attribute filters
- Retrieval and Filtering
- Test retrieval performance
- Apply attribute-based filters
- Evaluation and Analysis
- Compare predictions to ground truth
- Analyze performance metrics
Setup and Dependencies
Example Generations
Generating high-quality training and evaluation data for machine learning tasks can be costly and time-consuming. Synthetic data offers a practical and scalable alternative. In this notebook, the OpenAI Image API is used to generate synthetic images, while the Responses API is employed to create synthetic text, enabling efficient prototyping and experimentation across multimodal tasks.
Data Processing
In this example, we’ll work with a pre-generated synthetic dataset of customer feedback that includes short text snippets, images from customer reviews, and occasionally combined multimodal entries. You can also generate your own synthetic dataset using the examples provided above to tailor the data to your specific use case.
Populating Vector Store
This example uses OpenAI's built-in vector store and file search capabilities to build a RAG system that can analyse customer experiences from their feedback, which can be both visual and text-based. We create two vector stores for comparisons, one with image understanding and one without.
Retrieval and Filtering
We can analyse our dataset with natural language queries with the help of File Search. For the text-only dataset, we see that information is missing that could inform our analysis.
The only positive review for spaghetti in July has visual feedback and we can see the RAG system with only text based context available is uncertain about positive details. However with image context provided the second RAG system is able to provide a more accurate response.
We can confirm if this is correct by checking the retrieved images.
Likewise we can test this for negative reviews in June concerning any burnt pizza.
We can confirm if this is correct by checking the retrieved images.
Evaluation and Analysis
As our dataset likely evolves over time and we want to evaluate new models, we can use the OpenAI Evaluation API to evaluate the performance of our system for sentiment analysis. In this simple example, using the string_check criteria we checked if the output was one of the three possible values: positive, negative, or unclear.
We can retrieve the results of these evaluation runs and perform some local analysis. In this case, we will compare the performance of the text-only and text+image runs and evaluate how increasing the number of total tokens (through the addition of image context) affects the accuracy of the model. We can also do some basic error analysis by analysing the model input of the failed examples.
Finally, let's clean up some of the resources we created.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.