Process Data in Batches with OpenAI API

Batch processing with the Batch API

The new Batch API allows to create async batch jobs for a lower price and with higher rate limits.

Batches will be completed within 24h, but may be processed sooner depending on global usage.

Ideal use cases for the Batch API include:

• Tagging, captioning, or enriching content on a marketplace or blog
• Categorizing and suggesting answers for support tickets
• Performing sentiment analysis on large datasets of customer feedback
• Generating summaries or translations for collections of documents or articles

and much more!

This cookbook will walk you through how to use the Batch API with a couple of practical examples.

We will start with an example to categorize movies using gpt-4o-mini, and then cover how we can use the vision capabilities of this model to caption images.

Please note that multiple models are available through the Batch API, and that you can use the same parameters in your Batch API calls as with the Chat Completions endpoint.

Setup

First example: Categorizing movies

In this example, we will use gpt-4o-mini to extract movie categories from a description of the movie. We will also extract a 1-sentence summary from this description.

We will use JSON mode to extract categories as an array of strings and the 1-sentence summary in a structured format.

For each movie, we want to get a result that looks like this:

{
    categories: ['category1', 'category2', 'category3'],
    summary: '1-sentence summary'
}

Loading data

We will use the IMDB top 1000 movies dataset for this example.

Processing step

Here, we will prepare our requests by first trying them out with the Chat Completions endpoint.

Once we're happy with the results, we can move on to creating the batch file.

Creating the batch file

The batch file, in the jsonl format, should contain one line (json object) per request.
Each request is defined as such:

{
    "custom_id": <REQUEST_ID>,
    "method": "POST",
    "url": "/v1/chat/completions",
    "body": {
        "model": <MODEL>,
        "messages": <MESSAGES>,
        // other parameters
    }
}

Note: the request ID should be unique per batch. This is what you can use to match results to the initial input files, as requests will not be returned in the same order.

Uploading the file

Creating the batch job

Checking batch status

Note: this can take up to 24h, but it will usually be completed faster.

You can continue checking until the status is 'completed'.

Retrieving results

Reading results

Reminder: the results are not in the same order as in the input file.
Make sure to check the custom_id to match the results against the input requests

Second example: Captioning images

In this example, we will use gpt-4-turbo to caption images of furniture items.

We will use the vision capabilities of the model to analyze the images and generate the captions.

Loading data

We will use the Amazon furniture dataset for this example.

Processing step

Again, we will first prepare our requests with the Chat Completions endpoint, and create the batch file afterwards.

Creating the batch job

As with the first example, we will create an array of json tasks to generate a jsonl file and use it to create the batch job.

Getting results

As with the first example, we can retrieve results once the batch job is done.

Reminder: the results are not in the same order as in the input file.
Make sure to check the custom_id to match the results against the input requests

Wrapping up

In this cookbook, we have seen two examples of how to use the new Batch API, but keep in mind that the Batch API works the same way as the Chat Completions endpoint, supporting the same parameters and most of the recent models (gpt-4o, gpt-4o-mini, gpt-4-turbo, gpt-3.5-turbo...).

By using this API, you can significantly reduce costs, so we recommend switching every workload that can happen async to a batch job with this new API.