Processing and narrating a video with GPT-4.1-mini's visual capabilities and GPT-4o TTS API
This notebook demonstrates how to use GPT's visual capabilities with a video. Although GPT-4.1-mini doesn't take videos as input directly, we can use vision and the 1M token context window to describe the static frames of a whole video at once. We'll walk through two examples:
Get this prompt chain
Processing and narrating a video with GPT-4.1-mini's visual capabilities and GPT-4o TTS API
This notebook demonstrates how to use GPT's visual capabilities with a video. Although GPT-4.1-mini doesn't take videos as input directly, we can use vision and the 1M token context window to describe the static frames of a whole video at once. We'll walk through two examples:
- Using GPT-4.1-mini to get a description of a video
- Generating a voiceover for a video with GPT-4o TTS API
1. Using GPT's visual capabilities to get a description of a video
First, we use OpenCV to extract frames from a nature video containing bisons and wolves:
Display frames to make sure we've read them in correctly:
Once we have the video frames, we craft our prompt and send a request to GPT (Note that we don't need to send every frame for GPT to understand what's going on):
2. Generating a voiceover for a video with GPT-4.1 and the GPT-4o TTS API
Let's create a voiceover for this video in the style of David Attenborough. Using the same video frames we prompt GPT to give us a short script:
Now, we can work with the GPT-4o TTS model and provide it a set of instructions on how the voice should sound. You can play around with the voice models and instructers at OpenAI.fm. We can then pass in the script we generated above with GPT-4.1-mini and generate audio of the voiceover: